2024-09-18T11:32:53.640 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T11:32:53.657 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T11:32:53.736 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} email: skanta@redhat.com first_in_suite: false job_id: '7910973' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request mon_bind_msgr2: false sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T11:32:53.736 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T11:32:53.737 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T11:32:53.737 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T11:32:53.738 INFO:teuthology.task.internal:Checking packages... 2024-09-18T11:32:53.758 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T11:32:53.758 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T11:32:53.758 INFO:teuthology.packaging:ref: None 2024-09-18T11:32:53.758 INFO:teuthology.packaging:tag: None 2024-09-18T11:32:53.758 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:32:53.758 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:32:53.758 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:32:53.901 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:32:53.902 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T11:32:53.909 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T11:32:53.909 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T11:32:53.919 INFO:teuthology.task.internal:Saving configuration 2024-09-18T11:32:53.932 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T11:32:53.939 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T11:32:53.965 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 11:26:49.613313', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T11:32:53.989 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 11:26:49.614430', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T11:32:53.989 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T11:32:53.996 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-18T11:32:53.996 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-18T11:32:53.996 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T11:32:54.059 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f47ecdbbac0>, signals=[15]) 2024-09-18T11:32:54.059 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T11:32:54.065 INFO:teuthology.task.internal:Opening connections... 2024-09-18T11:32:54.065 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T11:32:54.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:32:54.152 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T11:32:54.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:32:54.239 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T11:32:54.246 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-09-18T11:32:54.253 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-09-18T11:32:54.253 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-09-18T11:32:54.302 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu" 2024-09-18T11:32:54.302 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="20.04" 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=focal 2024-09-18T11:32:54.303 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=focal 2024-09-18T11:32:54.304 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-09-18T11:32:54.328 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-09-18T11:32:54.335 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-09-18T11:32:54.335 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-09-18T11:32:54.384 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="Ubuntu" 2024-09-18T11:32:54.384 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:ID=ubuntu 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE=debian 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="20.04" 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_CODENAME=focal 2024-09-18T11:32:54.385 INFO:teuthology.orchestra.run.smithi129.stdout:UBUNTU_CODENAME=focal 2024-09-18T11:32:54.385 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-09-18T11:32:54.411 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T11:32:54.420 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T11:32:54.427 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T11:32:54.427 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-09-18T11:32:54.429 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-09-18T11:32:54.435 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T11:32:54.440 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T11:32:54.441 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-09-18T11:32:54.478 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-09-18T11:32:54.501 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T11:32:54.561 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T11:32:54.575 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T11:32:54.576 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T11:32:54.576 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T11:32:54.576 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T11:32:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:32:54.582 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-18T11:32:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:32:54.583 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-18T11:32:54.589 INFO:teuthology.orchestra.run.smithi112.stdout:5.4.0-124-generic 2024-09-18T11:32:54.589 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.4.0-124-generic 2024-09-18T11:32:54.589 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2024-09-18T11:32:54.632 INFO:teuthology.orchestra.run.smithi129.stdout:5.4.0-124-generic 2024-09-18T11:32:54.633 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.4.0-124-generic 2024-09-18T11:32:54.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2024-09-18T11:32:54.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-18T11:32:54.732 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-09-18T11:32:54.867 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T11:32:54.899 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:32:54.899 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T11:32:54.899 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T11:32:54.964 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T11:32:54.977 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:32:55.059 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T11:32:55.156 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T11:32:55.250 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T11:32:55.274 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T11:32:55.291 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T11:32:55.294 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T11:32:55.295 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T11:32:55.315 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T11:32:55.315 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T11:32:55.319 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T11:32:55.319 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T11:32:55.329 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T11:32:55.334 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T11:32:55.336 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T11:32:55.336 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T11:32:55.336 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T11:32:55.346 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T11:32:55.347 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T11:32:55.360 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T11:32:55.399 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T11:32:55.403 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T11:32:55.407 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T11:32:55.407 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T11:32:55.412 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T11:32:55.433 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T11:32:55.433 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T11:32:55.436 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T11:32:55.436 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T11:32:55.441 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T11:32:55.447 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T11:32:55.449 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T11:32:55.449 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T11:32:55.450 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T11:32:55.457 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T11:32:55.457 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T11:32:55.470 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T11:32:55.509 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T11:32:55.512 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T11:32:55.513 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T11:32:55.533 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T11:32:55.534 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T11:32:55.537 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T11:32:55.537 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T11:32:55.545 INFO:teuthology.orchestra.run.smithi129.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T11:32:55.550 INFO:teuthology.orchestra.run.smithi129.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T11:32:55.552 INFO:teuthology.orchestra.run.smithi129.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T11:32:55.552 INFO:teuthology.orchestra.run.smithi129.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T11:32:55.552 INFO:teuthology.orchestra.run.smithi129.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T11:32:55.553 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T11:32:55.561 INFO:teuthology.orchestra.run.smithi129.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T11:32:55.562 INFO:teuthology.orchestra.run.smithi129.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T11:32:55.562 INFO:teuthology.orchestra.run.smithi129.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T11:32:55.562 INFO:teuthology.orchestra.run.smithi129.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T11:32:55.578 INFO:teuthology.orchestra.run.smithi129.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T11:32:55.578 INFO:teuthology.orchestra.run.smithi129.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T11:32:55.578 INFO:teuthology.orchestra.run.smithi129.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T11:32:55.579 INFO:teuthology.orchestra.run.smithi129.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T11:32:55.939 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T11:32:55.971 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T11:32:55.983 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T11:32:55.983 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T11:32:55.984 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T11:32:56.080 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T11:32:56.093 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T11:32:56.093 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T11:32:56.134 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T11:32:56.152 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T11:32:56.157 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T11:32:56.157 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T11:32:56.160 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T11:32:56.160 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T11:32:56.187 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T11:32:56.187 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T11:32:56.188 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T11:32:56.269 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T11:32:56.269 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T11:32:56.269 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T11:32:56.270 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T11:32:59.643 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 22.0 MB in 3s (7,446 kB/s) 2024-09-18T11:32:59.689 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 22.0 MB in 3s (7,365 kB/s) 2024-09-18T11:33:01.245 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-09-18T11:33:01.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T11:33:01.276 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-18T11:33:01.304 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T11:33:01.346 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-09-18T11:33:01.390 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-18T11:33:01.565 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-09-18T11:33:01.565 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-09-18T11:33:01.610 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-18T11:33:01.611 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-18T11:33:01.766 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2024-09-18T11:33:01.766 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T11:33:01.766 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T11:33:01.766 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T11:33:01.767 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2024-09-18T11:33:01.767 INFO:teuthology.orchestra.run.smithi129.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T11:33:01.768 INFO:teuthology.orchestra.run.smithi129.stdout:Recommended packages: 2024-09-18T11:33:01.768 INFO:teuthology.orchestra.run.smithi129.stdout: thermald 2024-09-18T11:33:01.807 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2024-09-18T11:33:01.807 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T11:33:01.807 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T11:33:01.808 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T11:33:01.809 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2024-09-18T11:33:01.809 INFO:teuthology.orchestra.run.smithi112.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T11:33:01.809 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2024-09-18T11:33:01.809 INFO:teuthology.orchestra.run.smithi112.stdout: thermald 2024-09-18T11:33:01.828 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2024-09-18T11:33:01.828 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T11:33:01.828 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T11:33:01.828 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T11:33:01.830 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages will be upgraded: 2024-09-18T11:33:01.831 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T11:33:01.862 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2024-09-18T11:33:01.863 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T11:33:01.863 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T11:33:01.863 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T11:33:01.864 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2024-09-18T11:33:01.865 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T11:33:01.904 INFO:teuthology.orchestra.run.smithi129.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T11:33:01.904 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 77.2 MB of archives. 2024-09-18T11:33:01.905 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T11:33:01.905 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T11:33:02.049 INFO:teuthology.orchestra.run.smithi112.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T11:33:02.049 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 77.2 MB of archives. 2024-09-18T11:33:02.049 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T11:33:02.049 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T11:33:02.232 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T11:33:02.326 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T11:33:02.789 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T11:33:02.789 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T11:33:02.790 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T11:33:02.883 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T11:33:02.893 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T11:33:03.223 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T11:33:03.495 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T11:33:03.557 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 77.2 MB in 1s (73.1 MB/s) 2024-09-18T11:33:03.762 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T11:33:04.806 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T11:33:04.806 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T11:33:04.806 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T11:33:05.083 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T11:33:05.135 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T11:33:05.493 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T11:33:05.497 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:33:05.609 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:05.813 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 77.2 MB in 3s (23.6 MB/s) 2024-09-18T11:33:06.028 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T11:33:07.567 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T11:33:07.580 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:33:07.668 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:07.756 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T11:33:07.761 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:33:07.891 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:08.721 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T11:33:08.735 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:33:08.770 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:09.883 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T11:33:09.898 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:33:10.034 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:11.145 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T11:33:11.160 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:33:11.220 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:13.490 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:33:13.616 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:33:13.975 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:33:14.095 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:33:14.448 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T11:33:14.465 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T11:33:14.506 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T11:33:16.515 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:33:16.643 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:33:17.045 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:33:17.197 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:33:17.525 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T11:33:17.538 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T11:33:17.582 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T11:33:20.072 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T11:33:20.092 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:33:20.130 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:23.271 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T11:33:23.293 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:33:23.329 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:24.110 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:33:24.263 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:33:24.605 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T11:33:24.743 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:24.869 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T11:33:24.986 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:26.153 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:33:26.196 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:26.280 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:33:26.630 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T11:33:26.751 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:26.877 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T11:33:27.028 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:27.365 INFO:teuthology.orchestra.run.smithi129.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:33:27.365 INFO:teuthology.orchestra.run.smithi129.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T11:33:27.506 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:28.455 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:29.208 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T11:33:29.351 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T11:33:29.478 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:29.548 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T11:33:29.548 INFO:teuthology.orchestra.run.smithi129.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:33:29.615 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:33:29.615 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T11:33:29.774 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:30.942 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T11:33:31.068 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T11:33:31.211 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:33:31.282 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T11:33:31.282 INFO:teuthology.orchestra.run.smithi112.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:33:51.888 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T11:33:51.888 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub' 2024-09-18T11:33:51.896 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T11:33:51.922 INFO:teuthology.orchestra.run.smithi129.stdout:Generating grub configuration file ... 2024-09-18T11:33:52.248 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:33:52.271 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:33:52.663 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T11:33:52.674 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T11:33:53.301 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T11:33:53.302 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub' 2024-09-18T11:33:53.313 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T11:33:53.332 INFO:teuthology.orchestra.run.smithi112.stdout:Generating grub configuration file ... 2024-09-18T11:33:53.641 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:33:53.664 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:33:54.137 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T11:33:54.148 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T11:33:55.092 INFO:teuthology.orchestra.run.smithi129.stdout:done 2024-09-18T11:33:55.519 DEBUG:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2024-09-18T11:33:55.565 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2024-09-18T11:33:55.565 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2024-09-18T11:33:55.565 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2024-09-18T11:33:55.565 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2024-09-18T11:33:55.565 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 20 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 5.4.0.196.194 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2024-09-18T11:33:55.566 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2024-09-18T11:33:55.566 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T11:33:55.567 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T11:33:55.567 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T11:33:55.567 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-18T11:33:55.567 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T11:33:55.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2024-09-18T11:33:55.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-09-18T11:33:55.841 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:33:55.846 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T11:33:55.863 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T11:33:55.996 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T11:33:56.847 INFO:teuthology.orchestra.run.smithi112.stdout:done 2024-09-18T11:33:57.324 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2024-09-18T11:33:57.368 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 20 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.4.0.196.194 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2024-09-18T11:33:57.369 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2024-09-18T11:33:57.370 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2024-09-18T11:33:57.370 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T11:33:57.370 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T11:33:57.370 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T11:33:57.370 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T11:33:57.370 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T11:33:57.370 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2024-09-18T11:33:57.487 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-18T11:33:57.621 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T11:33:57.718 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-09-18T11:33:57.737 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:33:57.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T11:33:57.817 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-09-18T11:33:57.819 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T11:33:57.909 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T11:33:58.011 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-09-18T11:33:58.012 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-09-18T11:33:58.247 INFO:teuthology.orchestra.run.smithi129.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T11:33:58.247 INFO:teuthology.orchestra.run.smithi129.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T11:33:58.249 DEBUG:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2024-09-18T11:33:58.268 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2024-09-18T11:33:58.268 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2024-09-18T11:33:58.268 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2024-09-18T11:33:58.268 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2024-09-18T11:33:58.268 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 20 2024-09-18T11:33:58.268 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T11:33:58.268 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2024-09-18T11:33:58.269 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2024-09-18T11:33:58.269 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 5.4.0.196.194 2024-09-18T11:33:58.269 INFO:teuthology.orchestra.run.smithi129.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T11:33:58.269 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T11:33:58.269 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2024-09-18T11:33:58.269 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2024-09-18T11:33:58.269 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2024-09-18T11:33:58.269 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2024-09-18T11:33:58.269 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-18T11:33:58.316 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.4Ve54GsG5Q 2024-09-18T11:33:58.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub/grub.cfg /tmp/tmp.4Ve54GsG5Q 2024-09-18T11:33:58.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.4Ve54GsG5Q 2024-09-18T11:33:58.374 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.4Ve54GsG5Q is 9KB 2024-09-18T11:33:58.389 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.4Ve54GsG5Q 2024-09-18T11:33:58.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T11:33:58.454 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:33:58.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T11:33:58.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T11:33:58.489 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T11:33:58.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo update-grub 2024-09-18T11:33:59.466 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-18T11:33:59.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T11:33:59.562 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-18T11:33:59.795 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-18T11:33:59.795 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-18T11:33:59.899 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub' 2024-09-18T11:33:59.910 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T11:33:59.920 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-09-18T11:34:00.044 INFO:teuthology.orchestra.run.smithi112.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T11:34:00.044 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T11:34:00.046 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2024-09-18T11:34:00.067 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 20 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.4.0.196.194 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2024-09-18T11:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2024-09-18T11:34:00.069 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2024-09-18T11:34:00.069 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-18T11:34:00.074 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.5YsoQ2NUSH 2024-09-18T11:34:00.074 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5YsoQ2NUSH 2024-09-18T11:34:00.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.5YsoQ2NUSH 2024-09-18T11:34:00.171 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.5YsoQ2NUSH is 9KB 2024-09-18T11:34:00.180 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:34:00.188 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.5YsoQ2NUSH 2024-09-18T11:34:00.191 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:34:00.195 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T11:34:00.249 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:34:00.249 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T11:34:00.249 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T11:34:00.282 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T11:34:00.282 DEBUG:teuthology.orchestra.run.smithi112:> sudo update-grub 2024-09-18T11:34:00.578 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T11:34:00.589 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T11:34:01.709 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub' 2024-09-18T11:34:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T11:34:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-09-18T11:34:02.028 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:34:02.038 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:34:02.057 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-09-18T11:34:02.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-09-18T11:34:02.436 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T11:34:02.446 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T11:34:04.128 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-09-18T11:34:04.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-09-18T11:34:32.065 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-18T11:34:32.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-18T11:34:32.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:34:34.132 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T11:34:34.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-18T11:34:34.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:34:50.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-18T11:34:52.477 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-18T11:34:59.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-18T11:34:59.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:35:01.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-18T11:35:01.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:35:02.497 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-18T11:35:04.541 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-18T11:35:14.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-18T11:35:14.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:35:16.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-18T11:35:16.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:35:32.929 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-18T11:35:34.973 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-18T11:35:47.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-18T11:35:47.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:35:48.637 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-18T11:35:49.517 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-18T11:35:49.517 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-196-generic"... 2024-09-18T11:35:49.518 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-18T11:35:49.524 INFO:teuthology.orchestra.run.smithi129.stdout:5.4.0-196-generic 2024-09-18T11:35:49.525 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T11:35:49.525 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T11:35:49.525 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T11:35:49.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-18T11:35:49.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:35:50.392 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-18T11:35:50.525 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-18T11:35:50.526 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T11:35:50.699 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-09-18T11:35:50.731 DEBUG:teuthology.parallel:result is None 2024-09-18T11:35:51.307 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-18T11:35:51.307 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-18T11:35:51.307 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-18T11:35:51.313 INFO:teuthology.orchestra.run.smithi112.stdout:5.4.0-196-generic 2024-09-18T11:35:51.313 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T11:35:51.313 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T11:35:51.313 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T11:35:52.314 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T11:35:52.315 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T11:35:52.398 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-09-18T11:35:52.423 DEBUG:teuthology.parallel:result is None 2024-09-18T11:35:52.423 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T11:35:52.430 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T11:35:52.430 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T11:35:52.432 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T11:35:52.440 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T11:35:52.446 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T11:35:52.451 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T11:35:52.452 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T11:35:52.454 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T11:35:52.475 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T11:35:52.482 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T11:35:52.483 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T11:35:52.510 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T11:35:52.535 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:52.539 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:52.545 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:52.550 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:35:52.552 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T11:35:52.559 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T11:35:52.560 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T11:35:52.562 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T11:35:52.578 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T11:35:52.587 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T11:35:52.587 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T11:35:52.622 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T11:35:52.629 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T11:35:52.636 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T11:35:52.691 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:35:52.691 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T11:35:52.715 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T11:35:52.722 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T11:35:52.729 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:35:52.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T11:35:52.755 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-09-18T11:35:52.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-09-18T11:35:52.814 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T11:35:52.821 INFO:teuthology.task.internal:Starting timer... 2024-09-18T11:35:52.822 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T11:35:52.831 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T11:35:52.841 DEBUG:teuthology.task.selinux:Excluding smithi112: OS 'ubuntu' does not support SELinux 2024-09-18T11:35:52.841 DEBUG:teuthology.task.selinux:Excluding smithi129: OS 'ubuntu' does not support SELinux 2024-09-18T11:35:52.841 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T11:35:52.841 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-18T11:35:52.841 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T11:35:52.841 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T11:35:52.853 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T11:35:52.864 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T11:35:52.865 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T11:41:35.083 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-09-18T11:41:35.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-18T11:41:35.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:41:35.182 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-18T11:41:35.299 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-18T11:41:35.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-18T11:41:35.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:41:35.394 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-18T11:41:35.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-18T11:41:35.518 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T11:41:35.528 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T11:41:35.529 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T11:41:35.529 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T11:41:35.532 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T11:41:35.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T11:41:35.561 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T11:41:35.561 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: Command line: ntpd -gq 2024-09-18T11:41:35.562 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: proto: precision = 0.160 usec (-22) 2024-09-18T11:41:35.562 INFO:teuthology.orchestra.run.smithi112.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T11:41:35.563 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T11:41:35.563 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T11:41:35.563 INFO:teuthology.orchestra.run.smithi112.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T11:41:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T11:41:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T11:41:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T11:41:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: Listen normally on 3 enp3s0f1 172.21.15.112:123 2024-09-18T11:41:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: Listen normally on 4 lo [::1]:123 2024-09-18T11:41:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cb69%5]:123 2024-09-18T11:41:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:35 ntpd[13721]: Listening on routing socket on fd #22 for interface updates 2024-09-18T11:41:35.595 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T11:41:35.595 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: Command line: ntpd -gq 2024-09-18T11:41:35.596 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: proto: precision = 0.185 usec (-22) 2024-09-18T11:41:35.596 INFO:teuthology.orchestra.run.smithi129.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T11:41:35.597 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T11:41:35.597 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T11:41:35.597 INFO:teuthology.orchestra.run.smithi129.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T11:41:35.598 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T11:41:35.598 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T11:41:35.598 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T11:41:35.598 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: Listen normally on 3 enp3s0f1 172.21.15.129:123 2024-09-18T11:41:35.598 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: Listen normally on 4 lo [::1]:123 2024-09-18T11:41:35.598 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7261%5]:123 2024-09-18T11:41:35.598 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:35 ntpd[13731]: Listening on routing socket on fd #22 for interface updates 2024-09-18T11:41:36.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:36 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:36.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:36 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:36.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:36 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:36.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:36 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:36.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:36 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:36.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:36 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:36.595 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:36 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:36.595 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:36 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:37.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:37 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:37.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:37 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:37.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:37 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:37.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:37 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:37.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:37 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:37.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:37 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:37.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:37.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:37.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:37.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:37 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:38.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:38 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:38.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:38 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:38.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:38 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:38.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:38 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:38.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:38 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:38.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:38 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:38.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:38 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:38.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:38 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:38.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:38 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:38.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:38 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:38.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:38 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:38.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:38 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:38.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:38 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:38.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:38 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:39.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:39.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:39.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:39.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:39.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:39.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:39.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:39.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:39.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:39.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:39.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:39.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:39 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:39 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:40 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:40 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:40 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:40 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:40 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:40.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:40 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:40.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:40 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:40.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:40 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:40.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:40 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:40.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:40 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:40.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:40 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:40.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:40 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:41.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:41.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:41.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:41.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:41.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:41.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:41.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:41.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:41.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:41.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:41.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:41.564 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:41 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:41.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:41.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:41.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:41.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:41.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:41 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:42.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:42 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:42.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:42 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:42.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:42 ntpd[13721]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:42.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:42 ntpd[13721]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:42.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:42 ntpd[13721]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:42.562 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:42 ntpd[13721]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:42.563 INFO:teuthology.orchestra.run.smithi112.stdout:18 Sep 11:41:42 ntpd[13721]: ntpd: time slew -0.000494 s 2024-09-18T11:41:42.563 INFO:teuthology.orchestra.run.smithi112.stdout:ntpd: time slew -0.000494s 2024-09-18T11:41:42.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:42 ntpd[13721]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T11:41:42.563 INFO:teuthology.orchestra.run.smithi112.stderr:18 Sep 11:41:42 ntpd[13721]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T11:41:42.595 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:42 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:42.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:42 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:42.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:42 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:42.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:42 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:42.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:42 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:42.596 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:42 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T11:41:42.641 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2024-09-18T11:41:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:41:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:41:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:41:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:41:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:41:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:41:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:41:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:41:43.599 INFO:teuthology.orchestra.run.smithi129.stdout:18 Sep 11:41:43 ntpd[13731]: ntpd: time slew +0.000201 s 2024-09-18T11:41:43.599 INFO:teuthology.orchestra.run.smithi129.stdout:ntpd: time slew +0.000201s 2024-09-18T11:41:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T11:41:43.600 INFO:teuthology.orchestra.run.smithi129.stderr:18 Sep 11:41:43 ntpd[13731]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T11:41:43.684 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T11:41:43.685 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2024-09-18T11:41:43.685 INFO:teuthology.orchestra.run.smithi129.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:41:43.685 INFO:teuthology.orchestra.run.smithi129.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:41:43.685 INFO:teuthology.orchestra.run.smithi129.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:41:43.685 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:41:43.686 INFO:teuthology.run_tasks:Running task install... 2024-09-18T11:41:43.695 DEBUG:teuthology.task.install:project ceph 2024-09-18T11:41:43.695 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T11:41:43.695 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T11:41:43.695 INFO:teuthology.task.install:Using flavor: default 2024-09-18T11:41:43.701 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T11:41:43.701 INFO:teuthology.task.install:extra packages: [] 2024-09-18T11:41:43.701 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-key list | grep Ceph 2024-09-18T11:41:43.701 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-key list | grep Ceph 2024-09-18T11:41:43.802 INFO:teuthology.orchestra.run.smithi129.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T11:41:43.803 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T11:41:43.863 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T11:41:43.863 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T11:41:43.864 INFO:teuthology.orchestra.run.smithi129.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T11:41:43.864 INFO:teuthology.orchestra.run.smithi129.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T11:41:43.865 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T11:41:43.865 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:41:43.867 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T11:41:43.867 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:41:44.007 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T11:41:44.008 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:41:44.021 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T11:41:44.021 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:41:44.129 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:41:44.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T11:41:44.134 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:41:44.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T11:41:44.146 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-18T11:41:44.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-09-18T11:41:44.341 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T11:41:44.348 INFO:teuthology.orchestra.run.smithi129.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T11:41:44.363 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T11:41:44.369 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T11:41:44.387 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T11:41:44.393 INFO:teuthology.orchestra.run.smithi129.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T11:41:44.415 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T11:41:44.415 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:41:44.415 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T11:41:44.423 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T11:41:44.423 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:41:44.423 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T11:41:44.448 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T11:41:44.455 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T11:41:44.485 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T11:41:44.494 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T11:41:44.567 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T11:41:44.575 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T11:41:45.092 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 28.4 kB in 1s (37.1 kB/s) 2024-09-18T11:41:45.093 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 28.4 kB in 1s (36.5 kB/s) 2024-09-18T11:41:46.252 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-09-18T11:41:46.253 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-18T11:41:46.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:41:46.280 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:41:46.350 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-09-18T11:41:46.350 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-18T11:41:46.557 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-09-18T11:41:46.558 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-09-18T11:41:46.567 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-18T11:41:46.568 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-18T11:41:46.749 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T11:41:46.750 INFO:teuthology.orchestra.run.smithi129.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T11:41:46.750 INFO:teuthology.orchestra.run.smithi129.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T11:41:46.750 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2024-09-18T11:41:46.751 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T11:41:46.751 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T11:41:46.751 INFO:teuthology.orchestra.run.smithi129.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T11:41:46.751 INFO:teuthology.orchestra.run.smithi129.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T11:41:46.751 INFO:teuthology.orchestra.run.smithi129.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T11:41:46.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: socat xmlstarlet 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T11:41:46.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T11:41:46.754 INFO:teuthology.orchestra.run.smithi129.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T11:41:46.754 INFO:teuthology.orchestra.run.smithi129.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T11:41:46.754 INFO:teuthology.orchestra.run.smithi129.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T11:41:46.754 INFO:teuthology.orchestra.run.smithi129.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T11:41:46.754 INFO:teuthology.orchestra.run.smithi129.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T11:41:46.754 INFO:teuthology.orchestra.run.smithi129.stdout:Recommended packages: 2024-09-18T11:41:46.754 INFO:teuthology.orchestra.run.smithi129.stdout: btrfs-tools 2024-09-18T11:41:46.763 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T11:41:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T11:41:46.764 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T11:41:46.764 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2024-09-18T11:41:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T11:41:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T11:41:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: socat xmlstarlet 2024-09-18T11:41:46.768 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2024-09-18T11:41:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T11:41:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T11:41:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T11:41:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T11:41:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T11:41:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T11:41:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T11:41:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T11:41:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T11:41:46.769 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2024-09-18T11:41:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: btrfs-tools 2024-09-18T11:41:46.874 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2024-09-18T11:41:46.874 INFO:teuthology.orchestra.run.smithi129.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T11:41:46.874 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T11:41:46.874 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T11:41:46.875 INFO:teuthology.orchestra.run.smithi129.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T11:41:46.875 INFO:teuthology.orchestra.run.smithi129.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T11:41:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T11:41:46.877 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T11:41:46.877 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T11:41:46.877 INFO:teuthology.orchestra.run.smithi129.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T11:41:46.877 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T11:41:46.877 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T11:41:46.877 INFO:teuthology.orchestra.run.smithi129.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T11:41:46.877 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages will be upgraded: 2024-09-18T11:41:46.877 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 librbd1 2024-09-18T11:41:46.886 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2024-09-18T11:41:46.886 INFO:teuthology.orchestra.run.smithi112.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T11:41:46.886 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T11:41:46.886 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T11:41:46.888 INFO:teuthology.orchestra.run.smithi112.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T11:41:46.889 INFO:teuthology.orchestra.run.smithi112.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T11:41:46.890 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2024-09-18T11:41:46.890 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 librbd1 2024-09-18T11:41:46.943 INFO:teuthology.orchestra.run.smithi129.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T11:41:46.943 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 201 MB of archives. 2024-09-18T11:41:46.943 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T11:41:46.943 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T11:41:46.954 INFO:teuthology.orchestra.run.smithi112.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T11:41:46.954 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 201 MB of archives. 2024-09-18T11:41:46.954 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T11:41:46.954 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T11:41:47.026 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T11:41:47.027 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T11:41:47.028 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T11:41:47.035 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T11:41:47.037 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T11:41:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T11:41:47.039 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T11:41:47.044 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T11:41:47.048 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T11:41:47.054 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T11:41:47.068 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T11:41:47.069 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T11:41:47.074 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T11:41:47.078 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T11:41:47.079 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T11:41:47.085 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T11:41:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T11:41:47.108 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T11:41:47.113 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T11:41:47.115 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T11:41:47.116 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T11:41:47.116 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T11:41:47.118 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T11:41:47.118 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T11:41:47.118 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T11:41:47.119 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T11:41:47.119 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T11:41:47.123 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T11:41:47.124 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T11:41:47.126 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T11:41:47.126 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T11:41:47.128 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T11:41:47.129 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T11:41:47.129 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T11:41:47.130 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T11:41:47.133 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T11:41:47.138 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T11:41:47.139 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T11:41:47.140 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T11:41:47.141 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T11:41:47.144 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T11:41:47.144 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T11:41:47.144 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T11:41:47.145 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T11:41:47.146 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T11:41:47.148 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T11:41:47.149 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T11:41:47.149 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T11:41:47.150 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T11:41:47.151 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T11:41:47.152 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T11:41:47.152 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T11:41:47.154 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T11:41:47.154 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T11:41:47.158 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T11:41:47.159 INFO:teuthology.orchestra.run.smithi129.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T11:41:47.159 INFO:teuthology.orchestra.run.smithi129.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T11:41:47.168 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T11:41:47.169 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T11:41:47.175 INFO:teuthology.orchestra.run.smithi129.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T11:41:47.175 INFO:teuthology.orchestra.run.smithi129.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T11:41:47.176 INFO:teuthology.orchestra.run.smithi129.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T11:41:47.177 INFO:teuthology.orchestra.run.smithi129.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T11:41:47.178 INFO:teuthology.orchestra.run.smithi129.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T11:41:47.181 INFO:teuthology.orchestra.run.smithi129.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T11:41:47.183 INFO:teuthology.orchestra.run.smithi129.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T11:41:47.185 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T11:41:47.191 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T11:41:47.192 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T11:41:47.192 INFO:teuthology.orchestra.run.smithi129.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T11:41:47.192 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T11:41:47.194 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T11:41:47.194 INFO:teuthology.orchestra.run.smithi129.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T11:41:47.194 INFO:teuthology.orchestra.run.smithi129.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T11:41:47.195 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T11:41:47.197 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T11:41:47.198 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T11:41:47.202 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T11:41:47.203 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T11:41:47.209 INFO:teuthology.orchestra.run.smithi129.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T11:41:47.210 INFO:teuthology.orchestra.run.smithi129.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T11:41:47.210 INFO:teuthology.orchestra.run.smithi129.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T11:41:47.219 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T11:41:47.222 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T11:41:47.222 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T11:41:47.223 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T11:41:47.226 INFO:teuthology.orchestra.run.smithi129.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T11:41:47.257 INFO:teuthology.orchestra.run.smithi112.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T11:41:47.294 INFO:teuthology.orchestra.run.smithi129.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T11:41:47.304 INFO:teuthology.orchestra.run.smithi112.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T11:41:47.311 INFO:teuthology.orchestra.run.smithi129.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T11:41:47.312 INFO:teuthology.orchestra.run.smithi129.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T11:41:47.320 INFO:teuthology.orchestra.run.smithi112.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T11:41:47.322 INFO:teuthology.orchestra.run.smithi112.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T11:41:47.324 INFO:teuthology.orchestra.run.smithi129.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T11:41:47.324 INFO:teuthology.orchestra.run.smithi129.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T11:41:47.334 INFO:teuthology.orchestra.run.smithi112.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T11:41:47.386 INFO:teuthology.orchestra.run.smithi129.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T11:41:47.387 INFO:teuthology.orchestra.run.smithi129.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T11:41:47.388 INFO:teuthology.orchestra.run.smithi129.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T11:41:47.389 INFO:teuthology.orchestra.run.smithi129.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T11:41:47.390 INFO:teuthology.orchestra.run.smithi129.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T11:41:47.393 INFO:teuthology.orchestra.run.smithi129.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T11:41:47.394 INFO:teuthology.orchestra.run.smithi129.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T11:41:47.397 INFO:teuthology.orchestra.run.smithi112.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T11:41:47.398 INFO:teuthology.orchestra.run.smithi112.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T11:41:47.399 INFO:teuthology.orchestra.run.smithi112.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T11:41:47.401 INFO:teuthology.orchestra.run.smithi112.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T11:41:47.402 INFO:teuthology.orchestra.run.smithi129.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T11:41:47.402 INFO:teuthology.orchestra.run.smithi112.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T11:41:47.403 INFO:teuthology.orchestra.run.smithi112.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T11:41:47.403 INFO:teuthology.orchestra.run.smithi129.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T11:41:47.405 INFO:teuthology.orchestra.run.smithi112.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T11:41:47.406 INFO:teuthology.orchestra.run.smithi129.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T11:41:47.406 INFO:teuthology.orchestra.run.smithi112.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T11:41:47.407 INFO:teuthology.orchestra.run.smithi129.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T11:41:47.410 INFO:teuthology.orchestra.run.smithi129.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T11:41:47.413 INFO:teuthology.orchestra.run.smithi129.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T11:41:47.413 INFO:teuthology.orchestra.run.smithi129.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T11:41:47.414 INFO:teuthology.orchestra.run.smithi112.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T11:41:47.415 INFO:teuthology.orchestra.run.smithi129.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T11:41:47.415 INFO:teuthology.orchestra.run.smithi129.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T11:41:47.419 INFO:teuthology.orchestra.run.smithi129.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T11:41:47.420 INFO:teuthology.orchestra.run.smithi129.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T11:41:47.421 INFO:teuthology.orchestra.run.smithi129.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T11:41:47.422 INFO:teuthology.orchestra.run.smithi112.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T11:41:47.425 INFO:teuthology.orchestra.run.smithi112.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T11:41:47.426 INFO:teuthology.orchestra.run.smithi129.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T11:41:47.427 INFO:teuthology.orchestra.run.smithi112.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T11:41:47.427 INFO:teuthology.orchestra.run.smithi112.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T11:41:47.430 INFO:teuthology.orchestra.run.smithi112.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T11:41:47.430 INFO:teuthology.orchestra.run.smithi129.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T11:41:47.430 INFO:teuthology.orchestra.run.smithi112.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T11:41:47.431 INFO:teuthology.orchestra.run.smithi112.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T11:41:47.431 INFO:teuthology.orchestra.run.smithi112.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T11:41:47.432 INFO:teuthology.orchestra.run.smithi112.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T11:41:47.433 INFO:teuthology.orchestra.run.smithi112.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T11:41:47.435 INFO:teuthology.orchestra.run.smithi112.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T11:41:47.436 INFO:teuthology.orchestra.run.smithi112.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T11:41:47.438 INFO:teuthology.orchestra.run.smithi112.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T11:41:47.446 INFO:teuthology.orchestra.run.smithi129.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T11:41:47.449 INFO:teuthology.orchestra.run.smithi129.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T11:41:47.454 INFO:teuthology.orchestra.run.smithi112.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T11:41:47.457 INFO:teuthology.orchestra.run.smithi112.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T11:41:47.491 INFO:teuthology.orchestra.run.smithi129.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T11:41:47.492 INFO:teuthology.orchestra.run.smithi129.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T11:41:47.492 INFO:teuthology.orchestra.run.smithi129.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T11:41:47.492 INFO:teuthology.orchestra.run.smithi129.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T11:41:47.493 INFO:teuthology.orchestra.run.smithi129.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T11:41:47.493 INFO:teuthology.orchestra.run.smithi129.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T11:41:47.496 INFO:teuthology.orchestra.run.smithi112.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T11:41:47.497 INFO:teuthology.orchestra.run.smithi112.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T11:41:47.497 INFO:teuthology.orchestra.run.smithi112.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T11:41:47.497 INFO:teuthology.orchestra.run.smithi112.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T11:41:47.498 INFO:teuthology.orchestra.run.smithi112.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T11:41:47.498 INFO:teuthology.orchestra.run.smithi112.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T11:41:47.505 INFO:teuthology.orchestra.run.smithi129.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T11:41:47.505 INFO:teuthology.orchestra.run.smithi129.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T11:41:47.506 INFO:teuthology.orchestra.run.smithi129.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T11:41:47.508 INFO:teuthology.orchestra.run.smithi129.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T11:41:47.510 INFO:teuthology.orchestra.run.smithi112.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T11:41:47.511 INFO:teuthology.orchestra.run.smithi112.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T11:41:47.511 INFO:teuthology.orchestra.run.smithi112.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T11:41:47.513 INFO:teuthology.orchestra.run.smithi112.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T11:41:47.526 INFO:teuthology.orchestra.run.smithi129.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T11:41:47.526 INFO:teuthology.orchestra.run.smithi129.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T11:41:47.526 INFO:teuthology.orchestra.run.smithi129.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T11:41:47.528 INFO:teuthology.orchestra.run.smithi129.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T11:41:47.529 INFO:teuthology.orchestra.run.smithi129.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T11:41:47.530 INFO:teuthology.orchestra.run.smithi112.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T11:41:47.530 INFO:teuthology.orchestra.run.smithi112.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T11:41:47.531 INFO:teuthology.orchestra.run.smithi112.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T11:41:47.533 INFO:teuthology.orchestra.run.smithi112.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T11:41:47.534 INFO:teuthology.orchestra.run.smithi112.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T11:41:47.576 INFO:teuthology.orchestra.run.smithi129.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T11:41:47.578 INFO:teuthology.orchestra.run.smithi129.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T11:41:47.584 INFO:teuthology.orchestra.run.smithi129.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T11:41:47.598 INFO:teuthology.orchestra.run.smithi112.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T11:41:47.599 INFO:teuthology.orchestra.run.smithi112.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T11:41:47.607 INFO:teuthology.orchestra.run.smithi112.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T11:41:48.584 INFO:teuthology.orchestra.run.smithi112.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T11:41:48.617 INFO:teuthology.orchestra.run.smithi129.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T11:41:48.660 INFO:teuthology.orchestra.run.smithi112.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T11:41:48.662 INFO:teuthology.orchestra.run.smithi112.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T11:41:48.663 INFO:teuthology.orchestra.run.smithi112.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T11:41:48.664 INFO:teuthology.orchestra.run.smithi129.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T11:41:48.668 INFO:teuthology.orchestra.run.smithi129.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T11:41:48.671 INFO:teuthology.orchestra.run.smithi129.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T11:41:48.715 INFO:teuthology.orchestra.run.smithi112.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T11:41:48.723 INFO:teuthology.orchestra.run.smithi129.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T11:41:48.870 INFO:teuthology.orchestra.run.smithi112.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T11:41:48.880 INFO:teuthology.orchestra.run.smithi129.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T11:41:49.730 INFO:teuthology.orchestra.run.smithi129.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T11:41:49.731 INFO:teuthology.orchestra.run.smithi129.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T11:41:49.733 INFO:teuthology.orchestra.run.smithi112.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T11:41:49.734 INFO:teuthology.orchestra.run.smithi112.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T11:41:49.743 INFO:teuthology.orchestra.run.smithi129.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T11:41:49.747 INFO:teuthology.orchestra.run.smithi112.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T11:41:49.760 INFO:teuthology.orchestra.run.smithi129.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T11:41:49.761 INFO:teuthology.orchestra.run.smithi129.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T11:41:49.762 INFO:teuthology.orchestra.run.smithi129.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T11:41:49.764 INFO:teuthology.orchestra.run.smithi112.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T11:41:49.765 INFO:teuthology.orchestra.run.smithi112.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T11:41:49.766 INFO:teuthology.orchestra.run.smithi112.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T11:41:49.965 INFO:teuthology.orchestra.run.smithi129.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T11:41:49.981 INFO:teuthology.orchestra.run.smithi112.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T11:41:50.219 INFO:teuthology.orchestra.run.smithi129.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T11:41:50.219 INFO:teuthology.orchestra.run.smithi129.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T11:41:50.235 INFO:teuthology.orchestra.run.smithi112.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T11:41:50.235 INFO:teuthology.orchestra.run.smithi112.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T11:41:52.438 INFO:teuthology.orchestra.run.smithi112.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T11:41:52.443 INFO:teuthology.orchestra.run.smithi112.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T11:41:52.443 INFO:teuthology.orchestra.run.smithi112.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T11:41:52.522 INFO:teuthology.orchestra.run.smithi129.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T11:41:52.527 INFO:teuthology.orchestra.run.smithi129.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T11:41:52.527 INFO:teuthology.orchestra.run.smithi129.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T11:41:52.957 INFO:teuthology.orchestra.run.smithi112.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T11:41:53.099 INFO:teuthology.orchestra.run.smithi129.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T11:41:53.608 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 201 MB in 6s (33.2 MB/s) 2024-09-18T11:41:53.739 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 201 MB in 6s (32.4 MB/s) 2024-09-18T11:41:53.778 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T11:41:53.833 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T11:41:53.837 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T11:41:53.918 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T11:41:53.942 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T11:41:53.996 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T11:41:54.000 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T11:41:54.082 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T11:41:54.304 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:54.430 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T11:41:54.436 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T11:41:55.067 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:55.184 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T11:41:55.243 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:55.390 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T11:41:55.846 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T11:41:55.863 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:55.904 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:56.069 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T11:41:56.088 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:56.127 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:56.232 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rados. 2024-09-18T11:41:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:56.281 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:56.496 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rados. 2024-09-18T11:41:56.515 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:56.555 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:56.609 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T11:41:56.627 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:41:56.667 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:56.919 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T11:41:56.932 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T11:41:56.936 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:56.949 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:41:56.978 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:56.991 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:57.243 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T11:41:57.263 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:57.293 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:57.314 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T11:41:57.331 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:41:57.372 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:57.629 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T11:41:57.645 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:41:57.700 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T11:41:57.718 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T11:41:57.721 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:57.783 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T11:41:58.024 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T11:41:58.036 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T11:41:58.043 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T11:41:58.054 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:58.082 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T11:41:58.094 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:58.351 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T11:41:58.367 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:41:58.401 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:41:58.463 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T11:41:58.480 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T11:41:58.521 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T11:41:58.762 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T11:41:58.782 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T11:41:58.820 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T11:41:58.891 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T11:41:58.907 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T11:41:58.974 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T11:41:59.215 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T11:41:59.231 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T11:41:59.273 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T11:41:59.310 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T11:41:59.327 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T11:41:59.369 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T11:41:59.643 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T11:41:59.659 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T11:41:59.701 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T11:41:59.730 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T11:41:59.746 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T11:41:59.788 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T11:42:00.046 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T11:42:00.063 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T11:42:00.129 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T11:42:00.141 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T11:42:00.161 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T11:42:00.224 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T11:42:00.440 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T11:42:00.457 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T11:42:00.490 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T11:42:00.702 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T11:42:00.719 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:42:00.771 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:42:00.987 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T11:42:01.003 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:42:01.081 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:42:01.288 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T11:42:01.307 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:42:01.338 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:42:01.607 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T11:42:01.624 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:42:01.657 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:42:01.725 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T11:42:01.743 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:42:01.783 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:42:02.035 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T11:42:02.051 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:42:02.093 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:42:02.178 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T11:42:02.195 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T11:42:02.236 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T11:42:02.455 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T11:42:02.472 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T11:42:02.530 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T11:42:02.581 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librgw2. 2024-09-18T11:42:02.598 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:02.655 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:02.867 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librgw2. 2024-09-18T11:42:02.882 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:02.916 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:03.441 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T11:42:03.457 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:03.499 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:03.869 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T11:42:03.885 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:03.936 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:04.111 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liboath0. 2024-09-18T11:42:04.127 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T11:42:04.169 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T11:42:04.281 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liboath0. 2024-09-18T11:42:04.299 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T11:42:04.339 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T11:42:04.479 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T11:42:04.495 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:04.538 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:04.675 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T11:42:04.692 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:04.725 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:05.016 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-common. 2024-09-18T11:42:05.034 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:05.099 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:05.212 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-common. 2024-09-18T11:42:05.229 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:05.270 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:07.629 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-base. 2024-09-18T11:42:07.645 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:07.698 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:07.951 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-base. 2024-09-18T11:42:07.970 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:08.019 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:08.492 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T11:42:08.510 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T11:42:08.542 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T11:42:08.864 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T11:42:08.881 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T11:42:08.954 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T11:42:08.961 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T11:42:08.971 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T11:42:09.012 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T11:42:09.314 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T11:42:09.330 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T11:42:09.381 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T11:42:09.568 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T11:42:09.584 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T11:42:09.626 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T11:42:09.633 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T11:42:09.653 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:09.692 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:09.937 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T11:42:09.954 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T11:42:09.987 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T11:42:10.097 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T11:42:10.114 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T11:42:10.154 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T11:42:10.190 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T11:42:10.207 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:10.248 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:10.466 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T11:42:10.483 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T11:42:10.524 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T11:42:10.670 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T11:42:10.686 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T11:42:10.728 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T11:42:10.861 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T11:42:10.877 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T11:42:10.919 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T11:42:11.039 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T11:42:11.059 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T11:42:11.097 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T11:42:11.255 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T11:42:11.274 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T11:42:11.373 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T11:42:11.459 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T11:42:11.479 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T11:42:11.517 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T11:42:11.625 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T11:42:11.640 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T11:42:11.683 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T11:42:11.854 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T11:42:11.875 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T11:42:11.904 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T11:42:11.985 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T11:42:12.001 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T11:42:12.044 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T11:42:12.199 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T11:42:12.215 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T11:42:12.257 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T11:42:12.330 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-mako. 2024-09-18T11:42:12.345 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T11:42:12.404 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T11:42:12.585 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T11:42:12.601 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T11:42:12.643 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T11:42:12.724 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T11:42:12.740 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T11:42:12.784 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T11:42:12.972 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-mako. 2024-09-18T11:42:12.989 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T11:42:13.030 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T11:42:13.085 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T11:42:13.101 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T11:42:13.135 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T11:42:13.383 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T11:42:13.401 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T11:42:13.433 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T11:42:13.437 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webob. 2024-09-18T11:42:13.455 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T11:42:13.496 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T11:42:13.769 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T11:42:13.785 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T11:42:13.828 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T11:42:13.857 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T11:42:13.873 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T11:42:13.915 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T11:42:14.147 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-webob. 2024-09-18T11:42:14.165 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T11:42:14.226 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T11:42:14.231 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T11:42:14.246 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T11:42:14.284 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T11:42:14.559 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T11:42:14.579 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T11:42:14.617 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T11:42:14.637 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T11:42:14.653 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T11:42:14.699 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T11:42:14.945 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T11:42:14.961 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T11:42:15.003 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T11:42:15.070 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T11:42:15.087 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T11:42:15.128 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T11:42:15.364 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T11:42:15.383 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T11:42:15.423 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-paste. 2024-09-18T11:42:15.425 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T11:42:15.442 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T11:42:15.481 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T11:42:15.734 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T11:42:15.753 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T11:42:15.776 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T11:42:15.793 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T11:42:15.793 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T11:42:15.834 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T11:42:16.146 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-paste. 2024-09-18T11:42:16.162 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T11:42:16.164 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T11:42:16.177 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T11:42:16.204 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T11:42:16.220 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T11:42:16.506 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T11:42:16.524 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T11:42:16.525 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T11:42:16.540 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T11:42:16.564 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T11:42:16.574 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T11:42:16.859 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T11:42:16.878 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T11:42:16.902 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T11:42:16.917 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T11:42:16.920 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T11:42:16.960 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T11:42:17.271 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T11:42:17.280 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T11:42:17.292 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T11:42:17.296 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T11:42:17.330 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T11:42:17.338 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T11:42:17.658 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T11:42:17.674 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T11:42:17.693 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T11:42:17.709 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:17.716 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T11:42:17.753 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:18.084 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T11:42:18.098 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T11:42:18.103 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T11:42:18.115 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:18.146 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T11:42:18.156 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:18.487 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T11:42:18.504 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:18.569 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:18.667 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T11:42:18.683 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:18.725 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:18.924 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T11:42:18.941 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:18.982 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:20.286 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T11:42:20.306 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:20.344 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:20.385 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T11:42:20.401 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:20.443 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:21.255 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T11:42:21.271 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:21.313 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:22.489 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph. 2024-09-18T11:42:22.507 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:22.547 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:22.866 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T11:42:22.882 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:22.924 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:23.345 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T11:42:23.363 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:23.419 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:23.467 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph. 2024-09-18T11:42:23.487 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:23.525 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:23.870 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T11:42:23.885 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:23.914 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package cephadm. 2024-09-18T11:42:23.928 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:23.933 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:23.972 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:24.291 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T11:42:24.308 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T11:42:24.351 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T11:42:24.365 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T11:42:24.383 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:24.423 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:24.688 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T11:42:24.706 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:24.745 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:24.951 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package cephadm. 2024-09-18T11:42:24.967 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:25.009 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:25.149 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T11:42:25.167 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T11:42:25.207 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T11:42:25.429 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T11:42:25.445 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T11:42:25.459 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T11:42:25.476 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T11:42:25.487 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T11:42:25.526 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T11:42:25.778 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T11:42:25.794 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T11:42:25.823 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T11:42:25.843 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:25.853 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T11:42:25.881 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:26.172 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-routes. 2024-09-18T11:42:26.188 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T11:42:26.230 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T11:42:26.243 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T11:42:26.263 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T11:42:26.301 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T11:42:26.550 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T11:42:26.567 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:26.570 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T11:42:26.587 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T11:42:26.600 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:26.629 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T11:42:26.923 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T11:42:26.943 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T11:42:27.006 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T11:42:27.326 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-routes. 2024-09-18T11:42:27.342 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T11:42:27.384 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T11:42:27.737 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T11:42:27.753 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:27.795 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:27.955 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T11:42:27.971 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T11:42:28.013 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T11:42:28.332 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T11:42:28.348 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T11:42:28.392 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T11:42:28.710 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T11:42:28.727 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T11:42:28.768 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T11:42:29.151 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T11:42:29.171 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T11:42:29.201 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T11:42:29.537 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T11:42:29.556 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T11:42:29.595 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T11:42:29.941 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T11:42:29.957 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T11:42:29.999 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T11:42:30.011 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T11:42:30.028 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T11:42:30.069 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T11:42:30.598 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T11:42:30.613 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T11:42:30.673 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T11:42:31.034 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T11:42:31.051 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T11:42:31.092 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T11:42:31.242 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T11:42:31.261 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T11:42:31.300 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T11:42:31.750 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T11:42:31.770 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:31.808 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:31.862 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T11:42:31.880 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T11:42:31.920 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T11:42:32.341 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T11:42:32.360 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T11:42:32.399 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T11:42:32.903 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T11:42:32.922 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T11:42:32.960 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T11:42:33.108 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T11:42:33.127 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:33.165 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:33.288 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T11:42:33.304 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T11:42:33.346 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T11:42:33.658 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T11:42:33.678 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T11:42:33.716 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T11:42:34.010 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T11:42:34.030 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T11:42:34.068 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T11:42:34.269 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T11:42:34.288 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T11:42:34.328 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T11:42:34.405 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T11:42:34.421 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T11:42:34.463 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T11:42:34.630 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T11:42:34.648 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T11:42:34.688 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T11:42:34.799 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T11:42:34.819 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T11:42:34.857 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T11:42:35.034 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T11:42:35.052 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T11:42:35.083 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T11:42:35.185 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T11:42:35.201 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T11:42:35.244 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T11:42:35.361 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T11:42:35.379 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T11:42:35.419 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T11:42:35.773 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T11:42:35.792 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T11:42:35.831 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T11:42:35.947 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T11:42:35.964 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T11:42:36.005 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T11:42:36.185 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T11:42:36.203 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T11:42:36.243 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T11:42:36.317 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T11:42:36.335 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T11:42:36.374 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T11:42:36.505 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T11:42:36.522 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T11:42:36.554 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T11:42:36.694 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T11:42:36.711 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T11:42:36.752 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T11:42:36.942 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T11:42:36.961 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T11:42:37.000 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T11:42:37.303 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T11:42:37.323 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T11:42:37.361 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T11:42:37.443 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T11:42:37.460 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:37.501 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:37.707 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T11:42:37.727 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T11:42:37.765 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T11:42:37.829 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T11:42:37.845 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T11:42:37.889 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T11:42:38.252 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T11:42:38.271 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T11:42:38.310 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:42:38.403 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T11:42:38.421 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:38.461 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:38.580 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package jq. 2024-09-18T11:42:38.596 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T11:42:38.638 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:42:38.772 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T11:42:38.792 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T11:42:38.830 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T11:42:38.891 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package socat. 2024-09-18T11:42:38.907 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T11:42:38.948 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T11:42:39.217 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T11:42:39.234 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T11:42:39.252 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T11:42:39.270 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T11:42:39.275 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:42:39.310 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T11:42:39.553 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package jq. 2024-09-18T11:42:39.570 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T11:42:39.628 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:42:39.674 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-test. 2024-09-18T11:42:39.691 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:39.732 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:39.873 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package socat. 2024-09-18T11:42:39.890 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T11:42:39.931 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T11:42:40.269 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T11:42:40.288 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T11:42:40.327 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T11:42:40.974 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-test. 2024-09-18T11:42:40.990 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:41.031 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:44.347 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T11:42:44.366 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:44.405 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:44.719 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T11:42:44.736 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T11:42:44.777 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:42:45.263 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T11:42:45.280 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:45.321 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:45.558 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T11:42:45.575 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:42:45.607 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:45.726 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T11:42:45.744 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T11:42:45.802 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T11:42:45.921 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T11:42:45.940 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T11:42:46.004 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:42:46.131 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T11:42:46.147 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T11:42:46.189 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:42:46.569 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T11:42:46.588 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:46.627 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:47.047 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T11:42:47.063 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T11:42:47.123 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T11:42:47.460 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T11:42:47.477 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T11:42:47.518 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:42:47.755 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T11:42:47.773 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T11:42:47.812 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T11:42:48.132 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T11:42:48.151 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T11:42:48.190 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T11:42:48.493 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-attr. 2024-09-18T11:42:48.514 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T11:42:48.551 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T11:42:48.854 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T11:42:48.874 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T11:42:48.920 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T11:42:49.257 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T11:42:49.275 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T11:42:49.284 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T11:42:49.303 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T11:42:49.315 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T11:42:49.342 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T11:42:49.634 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T11:42:49.651 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T11:42:49.692 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T11:42:49.703 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T11:42:49.720 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T11:42:49.770 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T11:42:50.056 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-attr. 2024-09-18T11:42:50.073 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T11:42:50.114 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T11:42:50.129 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T11:42:50.145 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T11:42:50.187 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T11:42:50.442 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T11:42:50.459 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T11:42:50.501 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T11:42:50.532 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T11:42:50.549 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T11:42:50.590 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T11:42:50.845 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T11:42:50.865 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T11:42:50.901 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T11:42:50.903 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T11:42:50.919 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T11:42:50.959 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T11:42:51.257 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T11:42:51.274 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T11:42:51.295 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-py. 2024-09-18T11:42:51.313 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T11:42:51.331 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T11:42:51.353 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T11:42:51.700 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T11:42:51.717 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T11:42:51.758 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T11:42:51.826 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T11:42:51.845 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T11:42:51.884 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T11:42:52.061 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T11:42:52.077 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T11:42:52.119 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T11:42:52.213 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T11:42:52.232 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T11:42:52.288 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T11:42:52.422 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T11:42:52.439 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T11:42:52.480 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T11:42:52.599 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T11:42:52.617 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T11:42:52.657 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T11:42:52.842 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T11:42:52.860 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T11:42:52.900 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T11:42:53.035 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-py. 2024-09-18T11:42:53.053 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T11:42:53.093 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T11:42:53.178 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T11:42:53.194 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T11:42:53.236 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T11:42:53.455 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T11:42:53.473 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T11:42:53.513 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T11:42:53.850 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T11:42:53.866 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T11:42:53.908 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T11:42:53.940 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package radosgw. 2024-09-18T11:42:53.957 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:54.006 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:54.236 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T11:42:54.253 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T11:42:54.294 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T11:42:54.681 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T11:42:54.701 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T11:42:54.739 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T11:42:55.017 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T11:42:55.034 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T11:42:55.075 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T11:42:55.713 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T11:42:55.733 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:55.771 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:55.858 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package radosgw. 2024-09-18T11:42:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:55.916 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:56.061 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T11:42:56.368 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T11:42:56.519 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T11:42:56.645 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T11:42:56.863 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:57.408 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user cephadm....done 2024-09-18T11:42:57.572 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:42:57.703 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T11:42:57.714 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T11:42:57.721 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:42:57.761 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:58.066 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T11:42:58.089 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T11:42:58.358 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T11:42:58.367 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:42:58.493 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T11:42:58.525 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T11:42:58.619 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T11:42:58.652 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T11:42:58.862 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:42:58.954 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T11:42:59.348 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T11:42:59.434 INFO:teuthology.orchestra.run.smithi129.stdout:Adding system user cephadm....done 2024-09-18T11:42:59.612 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:42:59.642 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T11:42:59.739 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T11:42:59.944 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T11:43:00.098 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T11:43:00.228 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T11:43:00.383 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:43:00.537 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:00.542 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T11:43:00.669 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T11:43:00.847 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T11:43:00.973 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T11:43:00.997 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T11:43:01.282 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T11:43:01.457 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T11:43:01.583 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T11:43:01.709 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T11:43:01.750 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T11:43:02.043 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T11:43:02.085 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T11:43:02.319 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T11:43:02.378 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T11:43:02.645 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:02.738 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T11:43:02.955 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T11:43:03.047 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T11:43:03.081 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T11:43:03.332 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T11:43:03.398 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T11:43:03.658 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T11:43:03.691 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T11:43:03.817 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T11:43:03.984 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T11:43:04.219 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T11:43:04.285 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T11:43:04.411 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T11:43:04.520 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T11:43:04.537 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T11:43:04.829 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T11:43:04.838 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T11:43:05.164 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T11:43:05.256 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T11:43:05.457 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T11:43:05.625 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T11:43:05.799 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T11:43:05.993 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T11:43:06.101 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T11:43:06.153 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:06.460 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T11:43:06.586 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T11:43:06.745 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T11:43:06.787 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:43:06.921 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T11:43:07.047 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T11:43:07.281 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T11:43:07.432 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T11:43:07.440 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T11:43:07.567 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T11:43:07.800 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T11:43:07.868 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T11:43:08.185 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T11:43:08.202 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:43:08.311 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:08.328 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T11:43:08.454 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T11:43:08.654 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:43:08.788 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T11:43:08.789 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T11:43:09.057 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T11:43:09.140 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T11:43:09.266 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T11:43:09.358 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T11:43:09.392 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T11:43:09.484 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T11:43:09.660 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T11:43:09.785 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T11:43:09.961 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:43:10.086 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T11:43:10.114 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T11:43:10.240 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T11:43:10.404 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T11:43:10.584 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T11:43:10.867 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T11:43:10.868 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T11:43:11.123 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T11:43:11.161 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T11:43:11.249 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T11:43:11.287 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T11:43:11.584 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T11:43:11.840 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T11:43:11.944 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T11:43:12.069 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T11:43:12.147 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T11:43:12.215 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T11:43:12.457 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T11:43:12.555 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T11:43:12.848 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:12.855 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T11:43:12.974 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T11:43:13.109 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T11:43:13.235 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T11:43:13.553 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T11:43:13.946 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T11:43:14.072 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T11:43:14.198 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T11:43:14.438 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:14.541 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T11:43:14.564 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:43:14.715 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T11:43:14.842 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:14.993 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T11:43:15.092 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T11:43:15.401 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T11:43:15.677 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:43:15.803 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T11:43:16.163 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:16.289 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T11:43:16.457 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:16.583 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:43:16.607 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:16.708 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T11:43:16.733 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T11:43:17.077 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T11:43:17.110 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T11:43:17.403 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T11:43:17.671 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:43:17.830 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T11:43:18.115 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:18.165 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:18.241 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T11:43:18.315 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T11:43:18.409 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T11:43:18.535 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:43:18.659 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:18.660 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T11:43:18.810 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T11:43:18.953 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T11:43:19.079 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T11:43:19.144 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T11:43:19.380 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:19.523 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:19.752 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T11:43:19.752 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T11:43:20.217 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:20.225 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:20.351 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T11:43:20.428 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T11:43:20.477 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T11:43:20.602 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:43:20.735 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T11:43:20.745 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T11:43:21.046 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T11:43:21.055 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T11:43:21.197 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T11:43:21.408 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:21.493 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:21.534 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:21.642 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:21.660 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:21.786 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:21.884 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T11:43:21.884 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T11:43:21.912 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T11:43:22.239 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T11:43:22.343 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:22.540 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:22.545 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T11:43:22.667 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:22.793 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T11:43:22.872 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T11:43:23.190 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T11:43:23.576 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:23.702 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:23.704 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:23.828 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:23.946 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:24.105 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T11:43:24.257 INFO:teuthology.orchestra.run.smithi112.stdout:Adding group ceph....done 2024-09-18T11:43:24.407 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T11:43:24.676 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user ceph....done 2024-09-18T11:43:24.725 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:24.835 INFO:teuthology.orchestra.run.smithi112.stdout:Setting system user ceph properties....done 2024-09-18T11:43:24.850 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:24.853 INFO:teuthology.orchestra.run.smithi112.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T11:43:25.002 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T11:43:25.002 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T11:43:25.320 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T11:43:25.798 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:25.923 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:25.930 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:26.225 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:43:26.225 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:43:26.515 INFO:teuthology.orchestra.run.smithi129.stdout:Adding group ceph....done 2024-09-18T11:43:26.868 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:26.925 INFO:teuthology.orchestra.run.smithi129.stdout:Adding system user ceph....done 2024-09-18T11:43:27.102 INFO:teuthology.orchestra.run.smithi129.stdout:Setting system user ceph properties....done 2024-09-18T11:43:27.121 INFO:teuthology.orchestra.run.smithi129.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T11:43:27.263 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T11:43:27.575 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T11:43:27.625 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T11:43:28.031 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:28.087 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:28.157 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:28.322 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T11:43:28.323 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T11:43:28.431 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:43:28.431 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:43:28.772 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:29.002 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T11:43:29.002 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T11:43:29.076 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:29.448 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:29.450 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T11:43:29.919 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:29.945 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T11:43:29.945 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T11:43:30.142 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T11:43:30.142 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T11:43:30.383 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:30.543 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:30.579 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:30.677 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:30.805 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T11:43:30.805 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T11:43:30.877 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T11:43:30.877 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T11:43:31.280 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:31.345 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:31.488 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:31.614 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:31.782 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:31.789 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T11:43:31.789 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T11:43:31.917 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T11:43:32.069 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T11:43:32.249 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:32.370 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T11:43:32.376 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:32.518 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:32.747 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T11:43:32.747 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T11:43:33.203 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:33.356 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:33.471 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:33.656 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:43:33.783 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T11:43:33.934 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T11:43:34.219 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T11:43:35.182 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T11:43:35.800 INFO:teuthology.orchestra.run.smithi112.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T11:43:35.805 DEBUG:teuthology.parallel:result is None 2024-09-18T11:43:37.230 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T11:43:37.862 INFO:teuthology.orchestra.run.smithi129.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T11:43:37.866 DEBUG:teuthology.parallel:result is None 2024-09-18T11:43:37.867 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:43:38.009 DEBUG:teuthology.orchestra.run.smithi112:> dpkg-query -W -f '${Version}' ceph 2024-09-18T11:43:38.033 INFO:teuthology.orchestra.run.smithi112.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:43:38.033 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:43:38.034 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T11:43:38.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:43:38.175 DEBUG:teuthology.orchestra.run.smithi129:> dpkg-query -W -f '${Version}' ceph 2024-09-18T11:43:38.198 INFO:teuthology.orchestra.run.smithi129.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:43:38.199 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:43:38.199 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T11:43:38.201 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T11:43:38.201 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:38.201 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T11:43:38.218 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:43:38.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T11:43:38.233 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T11:43:38.234 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:38.234 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T11:43:38.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T11:43:38.301 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:43:38.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T11:43:38.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T11:43:38.332 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T11:43:38.332 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:38.332 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T11:43:38.348 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T11:43:38.358 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:43:38.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T11:43:38.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T11:43:38.428 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T11:43:38.429 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:38.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T11:43:38.445 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T11:43:38.502 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:43:38.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T11:43:38.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T11:43:38.573 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T11:43:38.650 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T11:43:38.651 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2024-09-18T11:43:38.654 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2024-09-18T11:43:38.669 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T11:43:38.669 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T11:43:38.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T11:43:38.682 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T11:43:38.682 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T11:43:38.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T11:43:38.750 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-18T11:43:38.750 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-18T11:43:38.750 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973', 'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7910973', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 4631, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'targets': {'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-18_09:13:00', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T11:43:38.751 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T11:43:38.753 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T11:43:38.760 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T11:43:38.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T11:43:38.818 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:38.818 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T11:43:38.825 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T11:43:38.825 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-09-18T11:43:38.832 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T11:43:38.832 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:43:38.833 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 351 Links: 1 2024-09-18T11:43:38.833 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:43:38.833 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-18 11:40:10.929158795 +0000 2024-09-18T11:43:38.833 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-18 11:40:10.925158888 +0000 2024-09-18T11:43:38.833 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-18 11:40:10.925158888 +0000 2024-09-18T11:43:38.833 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-18T11:43:38.833 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T11:43:38.888 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-18T11:43:38.888 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-18T11:43:38.888 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000419124 s, 1.2 MB/s 2024-09-18T11:43:38.889 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T11:43:38.898 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-09-18T11:43:38.905 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T11:43:38.905 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:43:38.905 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 359 Links: 1 2024-09-18T11:43:38.905 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:43:38.905 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-18 11:40:11.349149115 +0000 2024-09-18T11:43:38.906 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-18 11:40:11.345149206 +0000 2024-09-18T11:43:38.906 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-18 11:40:11.345149206 +0000 2024-09-18T11:43:38.906 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-18T11:43:38.906 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T11:43:38.921 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-18T11:43:38.921 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-18T11:43:38.921 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000417188 s, 1.2 MB/s 2024-09-18T11:43:38.922 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T11:43:38.930 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-09-18T11:43:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T11:43:38.978 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:43:38.978 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 366 Links: 1 2024-09-18T11:43:38.979 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:43:38.979 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-18 11:40:11.981134546 +0000 2024-09-18T11:43:38.979 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-18 11:40:11.805138604 +0000 2024-09-18T11:43:38.979 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-18 11:40:11.805138604 +0000 2024-09-18T11:43:38.979 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-18T11:43:38.979 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T11:43:38.990 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-18T11:43:38.990 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-18T11:43:38.990 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000408398 s, 1.3 MB/s 2024-09-18T11:43:38.992 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T11:43:39.000 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-09-18T11:43:39.050 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T11:43:39.050 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:43:39.050 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 371 Links: 1 2024-09-18T11:43:39.051 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:43:39.051 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-18 11:40:12.421124404 +0000 2024-09-18T11:43:39.051 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-18 11:40:12.209129292 +0000 2024-09-18T11:43:39.051 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-18 11:40:12.209129292 +0000 2024-09-18T11:43:39.051 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-18T11:43:39.051 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T11:43:39.062 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-18T11:43:39.062 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-18T11:43:39.062 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000370182 s, 1.4 MB/s 2024-09-18T11:43:39.063 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T11:43:39.071 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T11:43:39.072 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:43:39.072 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T11:43:39.079 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T11:43:39.079 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-09-18T11:43:39.086 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T11:43:39.086 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:43:39.086 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 371 Links: 1 2024-09-18T11:43:39.086 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:43:39.086 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-18 11:40:04.063468134 +0000 2024-09-18T11:43:39.086 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-18 11:40:03.879472373 +0000 2024-09-18T11:43:39.087 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-18 11:40:03.879472373 +0000 2024-09-18T11:43:39.087 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2024-09-18T11:43:39.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T11:43:39.098 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-18T11:43:39.098 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-18T11:43:39.098 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.00035665 s, 1.4 MB/s 2024-09-18T11:43:39.099 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T11:43:39.150 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-09-18T11:43:39.157 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T11:43:39.157 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:43:39.157 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 378 Links: 1 2024-09-18T11:43:39.157 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:43:39.157 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-18 11:40:04.519457628 +0000 2024-09-18T11:43:39.157 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-18 11:40:04.283463065 +0000 2024-09-18T11:43:39.157 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-18 11:40:04.283463065 +0000 2024-09-18T11:43:39.157 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2024-09-18T11:43:39.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T11:43:39.167 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-18T11:43:39.167 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-18T11:43:39.167 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000402131 s, 1.3 MB/s 2024-09-18T11:43:39.168 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T11:43:39.218 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-09-18T11:43:39.225 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T11:43:39.225 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:43:39.225 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 382 Links: 1 2024-09-18T11:43:39.226 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:43:39.226 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-18 11:40:04.763452008 +0000 2024-09-18T11:43:39.226 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-18 11:40:04.759452099 +0000 2024-09-18T11:43:39.226 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-18 11:40:04.759452099 +0000 2024-09-18T11:43:39.226 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2024-09-18T11:43:39.226 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T11:43:39.240 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-18T11:43:39.240 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-18T11:43:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000336508 s, 1.5 MB/s 2024-09-18T11:43:39.241 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T11:43:39.249 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-09-18T11:43:39.300 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T11:43:39.300 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:43:39.300 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 387 Links: 1 2024-09-18T11:43:39.300 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:43:39.301 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-18 11:40:05.427436709 +0000 2024-09-18T11:43:39.301 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-18 11:40:05.215441593 +0000 2024-09-18T11:43:39.301 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-18 11:40:05.215441593 +0000 2024-09-18T11:43:39.301 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2024-09-18T11:43:39.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T11:43:39.313 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-18T11:43:39.313 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-18T11:43:39.313 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000371188 s, 1.4 MB/s 2024-09-18T11:43:39.314 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T11:43:39.323 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T11:43:39.323 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T11:43:39.323 INFO:tasks.ceph:Generating config... 2024-09-18T11:43:39.324 INFO:tasks.ceph:[client] debug monc = 20 2024-09-18T11:43:39.324 INFO:tasks.ceph:[client] debug ms = 1 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] debug ms = 1 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] debug rados = 20 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-18T11:43:39.325 INFO:tasks.ceph:[global] ms type = async 2024-09-18T11:43:39.325 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-18T11:43:39.326 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-18T11:43:39.327 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T11:43:39.327 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T11:43:39.328 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-18T11:43:39.328 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-18T11:43:39.328 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-18T11:43:39.328 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-18T11:43:39.328 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T11:43:39.328 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-18T11:43:39.328 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-18T11:43:39.328 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T11:43:39.329 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-18T11:43:39.329 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-18T11:43:39.329 INFO:tasks.ceph:Setting up mon.a... 2024-09-18T11:43:39.329 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-18T11:43:39.364 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/ceph.keyring 2024-09-18T11:43:39.367 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-18T11:43:39.398 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T11:43:39.453 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.112:6789]'), ('mon.c', '[v1:172.21.15.112:6790]'), ('mon.b', '[v1:172.21.15.129:6789]')] 2024-09-18T11:43:39.454 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.112:6789],[v1:172.21.15.112:6790],[v1:172.21.15.129:6789]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-18T11:43:39.454 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:39.455 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T11:43:39.460 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.112:6789]' --addv c '[v1:172.21.15.112:6790]' --addv b '[v1:172.21.15.129:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = octopus 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 15 (octopus) 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:election_strategy: 1 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:43:39.500 INFO:teuthology.orchestra.run.smithi112.stdout:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:43:39.501 INFO:teuthology.orchestra.run.smithi112.stdout:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:43:39.501 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-18T11:43:39.502 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T11:43:39.550 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e68e10b8-0874-4c25-87a0-f38d9ca2e0d2... 2024-09-18T11:43:39.552 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T11:43:39.554 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.588 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T11:43:39.589 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T11:43:39.590 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2024-09-18T11:43:39.591 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = "[v1:172.21.15.112:6789],[v1:172.21.15.112:6790],[v1:172.21.15.129:6789]" 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: debug objecter = 20 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: debug rados = 20 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: mon client directed command retry = 5 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: mon election default strategy = 1 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: ms bind msgr2 = False 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay max = 1 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay probability = 0.005 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay type = mon 2024-09-18T11:43:39.592 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject internal delays = 0.002 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject socket failures = 2500 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: ms type = async 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2024-09-18T11:43:39.593 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.594 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore block size = 96636764160 2024-09-18T11:43:39.595 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression algorithm = zstd 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression mode = aggressive 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore fsck on mount = True 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore zero block detection = True 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluefs = 20 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluestore = 20 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: debug rocksdb = 10 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd full ratio = 0.9 2024-09-18T11:43:39.596 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd nearfull ratio = 0.8 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd class default list = * 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd class load list = * 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify cached snaps = True 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify missing on start = True 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd failsafe full ratio = 0.95 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock override recovery settings = True 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd objectstore = bluestore 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd op queue = debug_random 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: osd op queue cut off = debug_random 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2024-09-18T11:43:39.597 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: debug monc = 10 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-18T11:43:39.598 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T11:43:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: mon min osdmap epochs = 25 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune interval = 2 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune min = 15 2024-09-18T11:43:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune txsize = 2 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: mon scrub inject crc mismatch = 0.01 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: mon scrub inject missing keys = 0.05 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: mon scrub interval = 300 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on pool no app = False 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: paxos service trim min = 5 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2024-09-18T11:43:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T11:43:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T11:43:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: debug monc = 20 2024-09-18T11:43:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-18T11:43:39.602 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2024-09-18T11:43:39.602 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.c] 2024-09-18T11:43:39.602 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.b] 2024-09-18T11:43:39.603 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-18T11:43:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-18T11:43:39.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-18T11:43:39.606 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "[v1:172.21.15.112:6789],[v1:172.21.15.112:6790],[v1:172.21.15.129:6789]" 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-18T11:43:39.607 INFO:teuthology.orchestra.run.smithi129.stdout: debug objecter = 20 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: debug rados = 20 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: mon client directed command retry = 5 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: mon election default strategy = 1 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: ms bind msgr2 = False 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject delay max = 1 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject delay probability = 0.005 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject delay type = mon 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject internal delays = 0.002 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject socket failures = 2500 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: ms type = async 2024-09-18T11:43:39.608 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-18T11:43:39.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-18T11:43:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression algorithm = zstd 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression mode = aggressive 2024-09-18T11:43:39.611 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore zero block detection = True 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 20 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 20 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 10 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-18T11:43:39.612 INFO:teuthology.orchestra.run.smithi129.stdout: osd class default list = * 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: osd class load list = * 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify cached snaps = True 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify missing on start = True 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock override recovery settings = True 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: osd op queue = debug_random 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: osd op queue cut off = debug_random 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-18T11:43:39.613 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 10 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-18T11:43:39.614 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-18T11:43:39.615 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-18T11:43:39.616 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon min osdmap epochs = 25 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune interval = 2 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune min = 15 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune txsize = 2 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon scrub inject crc mismatch = 0.01 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon scrub inject missing keys = 0.05 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon scrub interval = 300 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on pool no app = False 2024-09-18T11:43:39.617 INFO:teuthology.orchestra.run.smithi129.stdout: paxos service trim min = 5 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 20 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-18T11:43:39.618 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-18T11:43:39.619 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-18T11:43:39.619 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-18T11:43:39.619 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-18T11:43:39.702 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T11:43:39.703 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:39.703 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-18T11:43:39.710 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:39.710 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-18T11:43:39.715 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T11:43:39.715 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:39.715 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T11:43:39.715 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T11:43:39.739 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:43:39.739 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:39.786 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T11:43:39.786 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:43:39.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T11:43:39.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T11:43:39.814 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:43:39.814 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:39.821 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T11:43:39.821 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T11:43:39.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T11:43:39.865 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T11:43:39.868 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T11:43:39.868 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T11:43:39.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-18T11:43:39.946 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-18T11:43:39.959 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T11:43:39.959 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T11:43:39.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-18T11:43:39.975 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T11:43:39.975 INFO:tasks.ceph:role: osd.0 2024-09-18T11:43:39.976 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T11:43:39.976 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T11:43:40.513 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:43:40.513 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:43:40.513 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:43:40.513 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2024-09-18T11:43:40.513 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:43:40.513 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:43:40.513 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:43:40.514 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:43:40.514 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:43:40.514 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:43:40.515 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-18T11:43:40.515 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-18T11:43:40.584 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-18T11:43:40.596 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:43:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:43:40.597 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-18T11:43:40.607 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T11:43:40.607 INFO:tasks.ceph:role: osd.1 2024-09-18T11:43:40.607 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T11:43:40.607 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T11:43:41.155 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:43:41.155 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:43:41.155 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:43:41.155 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2024-09-18T11:43:41.155 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:43:41.156 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:43:41.156 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:43:41.156 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:43:41.156 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:43:41.156 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:43:41.157 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-18T11:43:41.157 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-18T11:43:41.211 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-18T11:43:41.223 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:43:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:43:41.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-18T11:43:41.234 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T11:43:41.234 INFO:tasks.ceph:role: osd.2 2024-09-18T11:43:41.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T11:43:41.235 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T11:43:41.694 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:43:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:43:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:43:41.694 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2024-09-18T11:43:41.694 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:43:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:43:41.695 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:43:41.695 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:43:41.695 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:43:41.695 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:43:41.696 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-18T11:43:41.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-18T11:43:41.745 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-18T11:43:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:43:41.757 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:43:41.757 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:41.827+0000 7f208ce1a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-18T11:43:41.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:41.831+0000 7f208ce1a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-18T11:43:41.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:41.831+0000 7f208ce1a540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:43:41.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:41.831+0000 7f208ce1a540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:43:41.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:41.831+0000 7f208ce1a540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:43:41.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:41.835+0000 7f208ce1a540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-18T11:43:42.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:42.403+0000 7f208ce1a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:43.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:43.179+0000 7f208ce1a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:43.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:43.959+0000 7f208ce1a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:44.840 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-18T11:43:44.853 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:44.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:44.887+0000 7f06bfc4e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-18T11:43:44.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:44.887+0000 7f06bfc4e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-18T11:43:44.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:44.887+0000 7f06bfc4e540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:43:44.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:44.887+0000 7f06bfc4e540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:43:44.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:44.887+0000 7f06bfc4e540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:43:44.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:44.887+0000 7f06bfc4e540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-18T11:43:45.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:45.459+0000 7f06bfc4e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:46.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:46.247+0000 7f06bfc4e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:47.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:47.027+0000 7f06bfc4e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:47.884 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T11:43:47.944 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:47.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:47.979+0000 7fed9805e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-18T11:43:47.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:47.979+0000 7fed9805e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-18T11:43:47.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:47.979+0000 7fed9805e540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:43:47.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:47.979+0000 7fed9805e540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:43:47.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:47.979+0000 7fed9805e540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:43:47.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:47.979+0000 7fed9805e540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-18T11:43:48.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:48.543+0000 7fed9805e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:49.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:49.327+0000 7fed9805e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:43:50.123+0000 7fed9805e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:50.979 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-18T11:43:50.995 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-18T11:43:51.012 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T11:43:51.012 INFO:tasks.ceph:role: osd.3 2024-09-18T11:43:51.012 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T11:43:51.012 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:43:51.576 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:43:51.578 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-18T11:43:51.578 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-18T11:43:51.679 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-18T11:43:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:43:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:43:51.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-18T11:43:51.749 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T11:43:51.749 INFO:tasks.ceph:role: osd.4 2024-09-18T11:43:51.749 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T11:43:51.749 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T11:43:52.252 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:43:52.252 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:43:52.252 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:43:52.253 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 2024-09-18T11:43:52.253 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:43:52.253 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:43:52.253 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:43:52.253 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:43:52.253 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:43:52.253 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:43:52.254 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-18T11:43:52.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-18T11:43:52.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-18T11:43:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:43:52.294 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:43:52.295 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-18T11:43:52.354 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T11:43:52.354 INFO:tasks.ceph:role: osd.5 2024-09-18T11:43:52.354 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T11:43:52.355 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T11:43:52.828 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:43:52.829 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:43:52.831 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-18T11:43:52.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-18T11:43:52.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-18T11:43:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:43:52.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:43:52.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:52.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:52.990+0000 7f9f4a8fd540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-18T11:43:52.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:52.990+0000 7f9f4a8fd540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-18T11:43:52.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:52.990+0000 7f9f4a8fd540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T11:43:52.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:52.990+0000 7f9f4a8fd540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T11:43:52.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:52.990+0000 7f9f4a8fd540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T11:43:52.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:52.990+0000 7f9f4a8fd540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-18T11:43:53.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:53.550+0000 7f9f4a8fd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:54.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:54.330+0000 7f9f4a8fd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:55.110+0000 7f9f4a8fd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:55.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-18T11:43:55.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:56.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:56.058+0000 7fec92a4d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-18T11:43:56.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:56.058+0000 7fec92a4d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-18T11:43:56.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:56.058+0000 7fec92a4d540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T11:43:56.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:56.058+0000 7fec92a4d540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T11:43:56.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:56.058+0000 7fec92a4d540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T11:43:56.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:56.058+0000 7fec92a4d540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-18T11:43:56.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:56.618+0000 7fec92a4d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:57.406+0000 7fec92a4d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:58.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:58.202+0000 7fec92a4d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:43:59.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T11:43:59.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:43:59.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:59.106+0000 7f87f24cc540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-18T11:43:59.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:59.106+0000 7f87f24cc540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-18T11:43:59.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:59.106+0000 7f87f24cc540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T11:43:59.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:59.106+0000 7f87f24cc540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T11:43:59.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:59.106+0000 7f87f24cc540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T11:43:59.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:59.106+0000 7f87f24cc540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-18T11:43:59.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:43:59.678+0000 7f87f24cc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:44:00.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:00.462+0000 7f87f24cc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:44:01.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:01.250+0000 7f87f24cc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:44:02.113 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T11:44:02.169 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T11:44:02.170 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:44:02.170 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-18T11:44:02.186 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:44:02.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-18T11:44:02.198 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:44:02.199 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-18T11:44:02.208 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:44:02.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-18T11:44:02.225 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:44:02.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-18T11:44:02.234 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:44:02.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-18T11:44:02.293 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:44:02.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-18T11:44:02.303 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:44:02.303 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-18T11:44:02.310 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T11:44:02.310 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T11:44:02.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T11:44:02.356 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2024-09-18T11:44:02.356 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDtvOpmRGGCMRAA5MjIB0vGq2RdA71xdPhyow== 2024-09-18T11:44:02.356 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2024-09-18T11:44:02.356 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDwvOpmnJQcNRAAaOL0QWJN0+mhEnQGQ4fW0Q== 2024-09-18T11:44:02.356 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2024-09-18T11:44:02.356 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDzvOpm8ZmBOhAAxdXZfYuSvZEAuX6h37trPQ== 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDrvOpmmoaEMxAARj2yQqJCSMtJeWS9HLXuDg== 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.3] 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQD4vOpmj5QyOxAAgtzEgRHAGlqTxrpQ2iESfA== 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.4] 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQD8vOpmbpq6AxAAfZGcXzpjXovsKPZV+bf0SA== 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.5] 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQD/vOpmKKWPBhAAWHN8fkdAm9yAfHn86W/ayA== 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2024-09-18T11:44:02.357 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDrvOpm2CNNOBAAi6pcLvdf7kZXi90WHbCwDg== 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDtvOpmRGGCMRAA5MjIB0vGq2RdA71xdPhyow== 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDwvOpmnJQcNRAAaOL0QWJN0+mhEnQGQ4fW0Q== 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDzvOpm8ZmBOhAAxdXZfYuSvZEAuX6h37trPQ== 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDrvOpmmoaEMxAARj2yQqJCSMtJeWS9HLXuDg== 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQD4vOpmj5QyOxAAgtzEgRHAGlqTxrpQ2iESfA== 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.4] 2024-09-18T11:44:02.366 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQD8vOpmbpq6AxAAfZGcXzpjXovsKPZV+bf0SA== 2024-09-18T11:44:02.367 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.5] 2024-09-18T11:44:02.367 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQD/vOpmKKWPBhAAWHN8fkdAm9yAfHn86W/ayA== 2024-09-18T11:44:02.367 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-18T11:44:02.367 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDrvOpm2CNNOBAAi6pcLvdf7kZXi90WHbCwDg== 2024-09-18T11:44:02.368 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.414 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.460 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.548 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T11:44:02.551 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T11:44:02.595 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.644 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.688 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:44:02.732 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T11:44:02.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T11:44:02.826 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T11:44:02.826 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-18T11:44:02.840 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:44:02.892 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-18T11:44:02.904 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-18T11:44:02.964 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:44:03.015 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-18T11:44:03.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-18T11:44:03.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:44:03.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-18T11:44:03.111 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:44:03.113 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:44:03.117 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-18T11:44:03.118 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T11:44:03.118 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T11:44:03.120 INFO:tasks.ceph.mon.a:Started 2024-09-18T11:44:03.120 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T11:44:03.120 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T11:44:03.123 INFO:tasks.ceph.mon.c:Started 2024-09-18T11:44:03.123 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T11:44:03.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T11:44:03.126 INFO:tasks.ceph.mon.b:Started 2024-09-18T11:44:03.126 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-18T11:44:03.126 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-18T11:44:03.126 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-18T11:44:03.128 INFO:tasks.ceph.mgr.x:Started 2024-09-18T11:44:03.128 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T11:44:03.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd crush tunables default 2024-09-18T11:44:03.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient: get_monmap_and_config 2024-09-18T11:44:03.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient: build_initial_monmap 2024-09-18T11:44:03.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient: monmap: 2024-09-18T11:44:03.226 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:03.226 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:03.226 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:03.225494+0000 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:03.225494+0000 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 1 Processor -- start 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 1 -- start start 2024-09-18T11:44:03.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient: init 2024-09-18T11:44:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient(hunting): picked mon.noname-c con 0x7f8b1006a7f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient(hunting): picked mon.noname-a con 0x7f8b1009bcc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient(hunting): picked mon.noname-b con 0x7f8b1006b2d0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1009f990 con 0x7f8b1009bcc0 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1006c5d0 con 0x7f8b1006a7f0 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1006c8b0 con 0x7f8b1006b2d0 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b1879c700 10 monclient(hunting): authenticate will time out at 815.066577s 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b15d37700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8b1009bcc0 0x7f8b101343f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46270/0 (socket says 172.21.15.112:46270) 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.223+0000 7f8b15d37700 1 -- 172.21.15.112:0/2309857543 learned_addr learned my addr 172.21.15.112:0/2309857543 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.227+0000 7f8b15d37700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6789/0 conn(0x7f8b1009bcc0 legacy=0x7f8b101343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.227+0000 7f8b16d39700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6790/0 conn(0x7f8b1006b2d0 legacy=0x7f8b10137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:03.227+0000 7f8b16538700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.129:6789/0 conn(0x7f8b1006a7f0 legacy=0x7f8b1006abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:06.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 10 monclient(hunting): tick 2024-09-18T11:44:06.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 1 monclient(hunting): continuing hunt 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6790/0 conn(0x7f8b1006b2d0 legacy=0x7f8b10137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.129:6789/0 conn(0x7f8b1006a7f0 legacy=0x7f8b1006abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6789/0 conn(0x7f8b1009bcc0 legacy=0x7f8b101343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8af40038b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8b1009bcc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8b1006b2d0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1006ce70 con 0x7f8b1009bcc0 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af400f5b0 con 0x7f8af40038b0 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af4010810 con 0x7f8b1006b2d0 2024-09-18T11:44:06.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.223+0000 7f8b07fff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:06.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.227+0000 7f8b16538700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.129:6789/0 conn(0x7f8af40038b0 legacy=0x7f8af4005cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:06.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.227+0000 7f8b16d39700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6790/0 conn(0x7f8b1006b2d0 legacy=0x7f8af400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:06.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:06.227+0000 7f8b15d37700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6789/0 conn(0x7f8b1009bcc0 legacy=0x7f8af4009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:09.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 10 monclient(hunting): tick 2024-09-18T11:44:09.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 1 monclient(hunting): continuing hunt 2024-09-18T11:44:09.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:44:09.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6790/0 conn(0x7f8b1006b2d0 legacy=0x7f8af400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:09.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.129:6789/0 conn(0x7f8af40038b0 legacy=0x7f8af4005cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:09.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6789/0 conn(0x7f8b1009bcc0 legacy=0x7f8af4009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:09.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:09.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8af4010d70 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:09.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8af400f770 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:09.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8b1006b2d0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:09.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af4011ad0 con 0x7f8af4010d70 2024-09-18T11:44:09.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af4010540 con 0x7f8af400f770 2024-09-18T11:44:09.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af4003d30 con 0x7f8b1006b2d0 2024-09-18T11:44:09.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.223+0000 7f8b07fff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:09.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.227+0000 7f8b16d39700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6790/0 conn(0x7f8b1006b2d0 legacy=0x7f8af400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:09.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.227+0000 7f8b15d37700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.129:6789/0 conn(0x7f8af400f770 legacy=0x7f8af400fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:09.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.227+0000 7f8b16538700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6789/0 conn(0x7f8af4010d70 legacy=0x7f8af40111a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911022240 0 0) 0x7f8af4010540 con 0x7f8af400f770 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): my global_id is 4099 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8afc003570 con 0x7f8af400f770 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887766570 0 0) 0x7f8af4003d30 con 0x7f8b1006b2d0 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): my global_id is 4100 2024-09-18T11:44:09.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af4010540 con 0x7f8b1006b2d0 2024-09-18T11:44:09.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800010631 0 0) 0x7f8af4011ad0 con 0x7f8af4010d70 2024-09-18T11:44:09.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:09.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient(hunting): my global_id is 4101 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af4003d30 con 0x7f8af4010d70 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77621017 0 0) 0x7f8af4010540 con 0x7f8b1006b2d0 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.129:6789/0 conn(0x7f8af400f770 legacy=0x7f8af400fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6789/0 conn(0x7f8af4010d70 legacy=0x7f8af40111a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient: _finish_hunting 0 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 monclient: found mon.noname-b 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af4004f70 con 0x7f8b1006b2d0 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient: _finish_auth 0 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:09.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8b00001920 con 0x7f8b1006b2d0 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b1879c700 5 monclient: authenticate success, global_id 4100 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b1879c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b15536700 10 monclient: dump: 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:09.718 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:09.719 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:09.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.711+0000 7f8b1879c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:09.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b00001b30 con 0x7f8b1006b2d0 2024-09-18T11:44:09.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b15536700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:09.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b15536700 1 -- 172.21.15.112:0/2309857543 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8b00001f00 con 0x7f8b1006b2d0 2024-09-18T11:44:09.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:09.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 4 set_mon_vals no callback set 2024-09-18T11:44:09.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b15536700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:09.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b15536700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:09.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b15536700 10 monclient: dump: 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 10 monclient: shutdown 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 1 -- 172.21.15.112:0/2309857543 >> v1:172.21.15.112:6790/0 conn(0x7f8b1006b2d0 legacy=0x7f8af400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:09.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 1 -- 172.21.15.112:0/2309857543 shutdown_connections 2024-09-18T11:44:09.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 1 -- 172.21.15.112:0/2309857543 >> 172.21.15.112:0/2309857543 conn(0x7f8b100a2430 msgr2=0x7f8b1009e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:09.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 1 -- 172.21.15.112:0/2309857543 shutdown_connections 2024-09-18T11:44:09.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.715+0000 7f8b1879c700 1 -- 172.21.15.112:0/2309857543 wait complete. 2024-09-18T11:44:09.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient: build_initial_monmap 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient: monmap: 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 librados: starting msgr at 2024-09-18T11:44:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 librados: starting objecter 2024-09-18T11:44:09.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 Processor -- start 2024-09-18T11:44:09.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 -- start start 2024-09-18T11:44:09.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 librados: setting wanted keys 2024-09-18T11:44:09.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 librados: calling monclient init 2024-09-18T11:44:09.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient: init 2024-09-18T11:44:09.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:09.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:09.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient(hunting): picked mon.noname-c con 0x7f8af400f770 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:09.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient(hunting): picked mon.noname-a con 0x7f8af4010d70 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:09.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient(hunting): picked mon.noname-b con 0x7f8b1006b2d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1006eeb0 con 0x7f8af4010d70 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b101218e0 con 0x7f8b1006b2d0 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b10121600 con 0x7f8af400f770 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b1879c700 10 monclient(hunting): authenticate will time out at 821.563719s 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b16538700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8af400f770 0x7f8b101324a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45112/0 (socket says 172.21.15.112:45112) 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b16538700 1 -- 172.21.15.112:0/3107601253 learned_addr learned my addr 172.21.15.112:0/3107601253 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.719+0000 7f8b15d37700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8af4010d70 0x7f8b101417d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39308/0 (socket says 172.21.15.112:39308) 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b16d39700 1 -- 172.21.15.112:0/3107601253 >> v1:172.21.15.129:6789/0 conn(0x7f8b1006b2d0 legacy=0x7f8b10149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b16d39700 1 -- 172.21.15.112:0/3107601253 >> v1:172.21.15.129:6789/0 conn(0x7f8b1006b2d0 legacy=0x7f8b10149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b16d39700 1 --1- 172.21.15.112:0/3107601253 >> v1:172.21.15.129:6789/0 conn(0x7f8b1006b2d0 0x7f8b10149b80 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b16d39700 1 --1- 172.21.15.112:0/3107601253 >> v1:172.21.15.129:6789/0 conn(0x7f8b1006b2d0 0x7f8b10149b80 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-18T11:44:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.129:6789/0 2024-09-18T11:44:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b16538700 1 -- 172.21.15.112:0/3107601253 >> v1:172.21.15.112:6790/0 conn(0x7f8af400f770 legacy=0x7f8b101324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 client.?.objecter ms_handle_connect 0x7f8af400f770 2024-09-18T11:44:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b15d37700 1 -- 172.21.15.112:0/3107601253 >> v1:172.21.15.112:6789/0 conn(0x7f8af4010d70 legacy=0x7f8b101417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 client.?.objecter ms_handle_connect 0x7f8af4010d70 2024-09-18T11:44:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500811082 0 0) 0x7f8b10121600 con 0x7f8af400f770 2024-09-18T11:44:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 monclient(hunting): my global_id is 4106 2024-09-18T11:44:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af4010540 con 0x7f8af400f770 2024-09-18T11:44:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719668700 0 0) 0x7f8b1006eeb0 con 0x7f8af4010d70 2024-09-18T11:44:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 10 monclient(hunting): my global_id is 4107 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.723+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b10121600 con 0x7f8af4010d70 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2109655694 0 0) 0x7f8b10121600 con 0x7f8af4010d70 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b1006eeb0 con 0x7f8af4010d70 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8b08001e20 con 0x7f8af4010d70 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient(hunting): dump: 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 163810986 0 0) 0x7f8b1006eeb0 con 0x7f8af4010d70 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 >> v1:172.21.15.112:6790/0 conn(0x7f8af400f770 legacy=0x7f8b101324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 >> v1:172.21.15.129:6789/0 conn(0x7f8b1006b2d0 legacy=0x7f8b10149b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 monclient: found mon.a 2024-09-18T11:44:09.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10121ba0 con 0x7f8af4010d70 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 5 monclient: authenticate success, global_id 4107 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 10 monclient: _renew_subs 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b101331c0 con 0x7f8af4010d70 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 10 monclient: _renew_subs 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b10141ee0 con 0x7f8af4010d70 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 1 librados: init done 2024-09-18T11:44:09.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b08001fe0 con 0x7f8af4010d70 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8b08005000 con 0x7f8af4010d70 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 monclient: dump: 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:09.735 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f8b080053e0 con 0x7f8af4010d70 2024-09-18T11:44:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b14d35700 4 set_mon_vals no callback set 2024-09-18T11:44:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 2898405418 0 0) 0x7f8b080062c0 con 0x7f8af4010d70 2024-09-18T11:44:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 10 client.4107.objecter ms_dispatch 0x7f8b10000fd0 osd_map(1..1 src has 1..1) v4 2024-09-18T11:44:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-09-18T11:44:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-09-18T11:44:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b067fc700 20 client.4107.objecter dump_active .. 0 homeless 2024-09-18T11:44:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.727+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af80050b0 con 0x7f8af4010d70 2024-09-18T11:44:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.735+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7f8b08005830 con 0x7f8af4010d70 2024-09-18T11:44:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.735+0000 7f8b067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:09.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.735+0000 7f8b067fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:09.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.807+0000 7f8b1879c700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T11:44:09.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.807+0000 7f8b1879c700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T11:44:09.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.807+0000 7f8b1879c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:09.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:09.807+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8af8005360 con 0x7f8af4010d70 2024-09-18T11:44:09.920 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:09.914+0000 7fb4ba254000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T11:44:10.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.095+0000 7f8b067fc700 1 -- 172.21.15.112:0/3107601253 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f8b08006500 con 0x7f8af4010d70 2024-09-18T11:44:10.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.095+0000 7f8b067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T11:44:10.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.095+0000 7f8b067fc700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-18T11:44:10.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.095+0000 7f8b1879c700 10 librados: watch_flush enter 2024-09-18T11:44:10.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.095+0000 7f8b1879c700 10 librados: watch_flush exit 2024-09-18T11:44:10.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.095+0000 7f8b1879c700 20 client.4107.objecter shutdown clearing up homeless session... 2024-09-18T11:44:10.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.095+0000 7f8b1879c700 10 client.4107.objecter successfully canceled tick 2024-09-18T11:44:10.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.099+0000 7f8b1879c700 10 monclient: shutdown 2024-09-18T11:44:10.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.099+0000 7f8b1879c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:10.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.099+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 >> v1:172.21.15.112:6789/0 conn(0x7f8af4010d70 legacy=0x7f8b101417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.099+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 shutdown_connections 2024-09-18T11:44:10.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.099+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 >> 172.21.15.112:0/3107601253 conn(0x7f8b100a2430 msgr2=0x7f8b10097160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:10.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.099+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 shutdown_connections 2024-09-18T11:44:10.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.099+0000 7f8b1879c700 1 -- 172.21.15.112:0/3107601253 wait complete. 2024-09-18T11:44:10.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:10.099+0000 7f8b1879c700 1 librados: shutdown 2024-09-18T11:44:10.102 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2024-09-18T11:44:10.115 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-18T11:44:10.115 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:44:10.116 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-18T11:44:10.131 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:44:10.132 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-18T11:44:10.191 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-18T11:44:10.191 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-18T11:44:10.201 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:44:10.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-18T11:44:10.215 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:44:10.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-18T11:44:10.275 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:44:10.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-18T11:44:10.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new ecb0abea-5208-453b-86e2-58b64b0634ab 0 2024-09-18T11:44:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.441+0000 7f71e91d2700 10 monclient: get_monmap_and_config 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.441+0000 7f71e91d2700 10 monclient: build_initial_monmap 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.441+0000 7f71e91d2700 10 monclient: monmap: 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:44:10.447434+0000 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:44:10.447434+0000 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:10.449 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.441+0000 7f71e91d2700 1 Processor -- start 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.441+0000 7f71e91d2700 1 -- start start 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.441+0000 7f71e91d2700 10 monclient: init 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient(hunting): picked mon.noname-b con 0x7f71e40a3820 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient(hunting): picked mon.noname-c con 0x7f71e40a4300 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient(hunting): picked mon.noname-a con 0x7f71e4097a50 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4096f90 con 0x7f71e4097a50 2024-09-18T11:44:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4097270 con 0x7f71e40a4300 2024-09-18T11:44:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4097e70 con 0x7f71e40a3820 2024-09-18T11:44:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient(hunting): authenticate will time out at 824.104218s 2024-09-18T11:44:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e37fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f71e40a4300 0x7f71e4137420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58164/0 (socket says 172.21.15.129:58164) 2024-09-18T11:44:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e37fe700 1 -- 172.21.15.129:0/3307485666 learned_addr learned my addr 172.21.15.129:0/3307485666 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:10.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e37fe700 1 -- 172.21.15.129:0/3307485666 >> v1:172.21.15.129:6789/0 conn(0x7f71e40a4300 legacy=0x7f71e4137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398528586 0 0) 0x7f71e4097270 con 0x7f71e40a4300 2024-09-18T11:44:10.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient(hunting): my global_id is 4105 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e3fff700 1 -- 172.21.15.129:0/3307485666 >> v1:172.21.15.112:6790/0 conn(0x7f71e40a3820 legacy=0x7f71e40a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71cc003570 con 0x7f71e40a4300 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e89d1700 1 -- 172.21.15.129:0/3307485666 >> v1:172.21.15.112:6789/0 conn(0x7f71e4097a50 legacy=0x7f71e413ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458665597 0 0) 0x7f71cc003570 con 0x7f71e40a4300 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 >> v1:172.21.15.112:6790/0 conn(0x7f71e40a3820 legacy=0x7f71e40a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 >> v1:172.21.15.112:6789/0 conn(0x7f71e4097a50 legacy=0x7f71e413ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 monclient: found mon.noname-c 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e4096cb0 con 0x7f71e40a4300 2024-09-18T11:44:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 5 monclient: authenticate success, global_id 4105 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f71d4001a00 con 0x7f71e40a4300 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: dump: 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71d4001be0 con 0x7f71e40a4300 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 1 -- 172.21.15.129:0/3307485666 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f71d4004a90 con 0x7f71e40a4300 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 4 set_mon_vals no callback set 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e2ffd700 10 monclient: dump: 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 10 monclient: shutdown 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:10.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.445+0000 7f71e91d2700 1 -- 172.21.15.129:0/3307485666 >> v1:172.21.15.129:6789/0 conn(0x7f71e40a4300 legacy=0x7f71e4137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 -- 172.21.15.129:0/3307485666 shutdown_connections 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 -- 172.21.15.129:0/3307485666 >> 172.21.15.129:0/3307485666 conn(0x7f71e4067680 msgr2=0x7f71e4067a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 -- 172.21.15.129:0/3307485666 shutdown_connections 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 -- 172.21.15.129:0/3307485666 wait complete. 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient: build_initial_monmap 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient: monmap: 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:10.461 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.463 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 librados: starting msgr at 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 librados: starting objecter 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 Processor -- start 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 -- start start 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 librados: setting wanted keys 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 librados: calling monclient init 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient: init 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient(hunting): picked mon.noname-b con 0x7f71e4097a50 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient(hunting): picked mon.noname-a con 0x7f71e40a3820 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient(hunting): picked mon.noname-c con 0x7f71e40a4300 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4122750 con 0x7f71e40a3820 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e413dae0 con 0x7f71e4097a50 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4120270 con 0x7f71e40a4300 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e91d2700 10 monclient(hunting): authenticate will time out at 824.110074s 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e3fff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f71e4097a50 0x7f71e413d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58176/0 (socket says 172.21.15.129:58176) 2024-09-18T11:44:10.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.449+0000 7f71e3fff700 1 -- 172.21.15.129:0/699645814 learned_addr learned my addr 172.21.15.129:0/699645814 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:10.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e3fff700 1 -- 172.21.15.129:0/699645814 >> v1:172.21.15.129:6789/0 conn(0x7f71e4097a50 legacy=0x7f71e413d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 client.?.objecter ms_handle_connect 0x7f71e4097a50 2024-09-18T11:44:10.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069639724 0 0) 0x7f71e413dae0 con 0x7f71e4097a50 2024-09-18T11:44:10.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): my global_id is 4108 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d0004f10 con 0x7f71e4097a50 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e89d1700 1 -- 172.21.15.129:0/699645814 >> v1:172.21.15.112:6790/0 conn(0x7f71e40a4300 legacy=0x7f71e4149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 client.?.objecter ms_handle_connect 0x7f71e40a4300 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 client.?.objecter ms_handle_connect 0x7f71e40a3820 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e37fe700 1 -- 172.21.15.129:0/699645814 >> v1:172.21.15.112:6789/0 conn(0x7f71e40a3820 legacy=0x7f71e4146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436837945 0 0) 0x7f71d0004f10 con 0x7f71e4097a50 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71e413dae0 con 0x7f71e4097a50 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f71d8001d00 con 0x7f71e4097a50 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): dump: 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535455675 0 0) 0x7f71e4120270 con 0x7f71e40a4300 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient(hunting): my global_id is 4112 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d0004f10 con 0x7f71e40a4300 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 226481732 0 0) 0x7f71e413dae0 con 0x7f71e4097a50 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 >> v1:172.21.15.112:6790/0 conn(0x7f71e40a4300 legacy=0x7f71e4149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 >> v1:172.21.15.112:6789/0 conn(0x7f71e40a3820 legacy=0x7f71e4146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 monclient: found mon.b 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e4122470 con 0x7f71e4097a50 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient: _finish_auth 0 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 5 monclient: authenticate success, global_id 4108 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 10 monclient: _renew_subs 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e413e7a0 con 0x7f71e4097a50 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 10 monclient: _renew_subs 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e4121740 con 0x7f71e4097a50 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 1 librados: init done 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71d80022b0 con 0x7f71e4097a50 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f71d8002640 con 0x7f71e4097a50 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e1ffb700 4 set_mon_vals no callback set 2024-09-18T11:44:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 monclient: dump: 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f71d8002aa0 con 0x7f71e4097a50 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (unknown 1757439615 0 0) 0x7f71d8006300 con 0x7f71e4097a50 2024-09-18T11:44:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 10 client.4108.objecter ms_dispatch 0x7f71e4000fd0 osd_map(2..2 src has 1..2) v4 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 3 client.4108.objecter handle_osd_map got epochs [2,2] > 0 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 3 client.4108.objecter handle_osd_map decoding full epoch 2 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71cbfff700 20 client.4108.objecter dump_active .. 0 homeless 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.453+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c40050b0 con 0x7f71e4097a50 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.457+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7f71d80058c0 con 0x7f71e4097a50 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.457+0000 7f71cbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.457+0000 7f71cbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.537+0000 7f71e91d2700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ecb0abea-5208-453b-86e2-58b64b0634ab", "id": 0}] 2024-09-18T11:44:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.537+0000 7f71e91d2700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ecb0abea-5208-453b-86e2-58b64b0634ab", "id": 0}] 2024-09-18T11:44:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.537+0000 7f71e91d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.537+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ecb0abea-5208-453b-86e2-58b64b0634ab", "id": 0} v 0) v1 -- 0x7f71c40058e0 con 0x7f71e4097a50 2024-09-18T11:44:10.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71cbfff700 1 -- 172.21.15.129:0/699645814 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ecb0abea-5208-453b-86e2-58b64b0634ab", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2382159413 0 732707515) 0x7f71d80065b0 con 0x7f71e4097a50 2024-09-18T11:44:10.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71cbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ecb0abea-5208-453b-86e2-58b64b0634ab", "id": 0}] 2024-09-18T11:44:10.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71cbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:10.550 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-09-18T11:44:10.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 10 librados: watch_flush enter 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 10 librados: watch_flush exit 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 20 client.4108.objecter shutdown clearing up homeless session... 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 10 client.4108.objecter successfully canceled tick 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 10 monclient: shutdown 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 >> v1:172.21.15.129:6789/0 conn(0x7f71e4097a50 legacy=0x7f71e413d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 shutdown_connections 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 >> 172.21.15.129:0/699645814 conn(0x7f71e4067680 msgr2=0x7f71e40a1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 shutdown_connections 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 1 -- 172.21.15.129:0/699645814 wait complete. 2024-09-18T11:44:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.545+0000 7f71e91d2700 1 librados: shutdown 2024-09-18T11:44:10.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 3164789a-58af-4cf7-bd86-260d70dd02c9 1 2024-09-18T11:44:10.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.665+0000 7fb9bde90700 10 monclient: get_monmap_and_config 2024-09-18T11:44:10.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.665+0000 7fb9bde90700 10 monclient: build_initial_monmap 2024-09-18T11:44:10.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.665+0000 7fb9bde90700 10 monclient: monmap: 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:44:10.671925+0000 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:44:10.671925+0000 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:10.674 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 1 Processor -- start 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 1 -- start start 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 10 monclient: init 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 10 monclient(hunting): picked mon.noname-b con 0x7fb9b81322a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 10 monclient(hunting): picked mon.noname-a con 0x7fb9b809e870 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 10 monclient(hunting): picked mon.noname-c con 0x7fb9b806e7e0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b80a47e0 con 0x7fb9b809e870 2024-09-18T11:44:10.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b809ec70 con 0x7fb9b806e7e0 2024-09-18T11:44:10.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b806ebe0 con 0x7fb9b81322a0 2024-09-18T11:44:10.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:10.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bde90700 10 monclient(hunting): authenticate will time out at 824.329374s 2024-09-18T11:44:10.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bd68f700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb9b806e7e0 0x7fb9b813aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58190/0 (socket says 172.21.15.129:58190) 2024-09-18T11:44:10.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bd68f700 1 -- 172.21.15.129:0/1339404393 learned_addr learned my addr 172.21.15.129:0/1339404393 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:10.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bd68f700 1 -- 172.21.15.129:0/1339404393 >> v1:172.21.15.129:6789/0 conn(0x7fb9b806e7e0 legacy=0x7fb9b813aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9bce8e700 1 -- 172.21.15.129:0/1339404393 >> v1:172.21.15.112:6790/0 conn(0x7fb9b81322a0 legacy=0x7fb9b8132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.669+0000 7fb9b7fff700 1 -- 172.21.15.129:0/1339404393 >> v1:172.21.15.112:6789/0 conn(0x7fb9b809e870 legacy=0x7fb9b8137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409860985 0 0) 0x7fb9b809ec70 con 0x7fb9b806e7e0 2024-09-18T11:44:10.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): my global_id is 4111 2024-09-18T11:44:10.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a8003570 con 0x7fb9b806e7e0 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736076717 0 0) 0x7fb9b806ebe0 con 0x7fb9b81322a0 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): my global_id is 4115 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b809ec70 con 0x7fb9b81322a0 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131044821 0 0) 0x7fb9b80a47e0 con 0x7fb9b809e870 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient(hunting): my global_id is 4116 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b806ebe0 con 0x7fb9b809e870 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2009257086 0 0) 0x7fb9a8003570 con 0x7fb9b806e7e0 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 >> v1:172.21.15.112:6790/0 conn(0x7fb9b81322a0 legacy=0x7fb9b8132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 >> v1:172.21.15.112:6789/0 conn(0x7fb9b809e870 legacy=0x7fb9b8137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 monclient: found mon.noname-c 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b806f120 con 0x7fb9b806e7e0 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb9b0001cc0 con 0x7fb9b806e7e0 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 5 monclient: authenticate success, global_id 4111 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: dump: 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9b0001f50 con 0x7fb9b806e7e0 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:10.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 1 -- 172.21.15.129:0/1339404393 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb9b0004e10 con 0x7fb9b806e7e0 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 4 set_mon_vals no callback set 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9b77fe700 10 monclient: dump: 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient: shutdown 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- 172.21.15.129:0/1339404393 >> v1:172.21.15.129:6789/0 conn(0x7fb9b806e7e0 legacy=0x7fb9b813aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- 172.21.15.129:0/1339404393 shutdown_connections 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- 172.21.15.129:0/1339404393 >> 172.21.15.129:0/1339404393 conn(0x7fb9b8068110 msgr2=0x7fb9b80a3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- 172.21.15.129:0/1339404393 shutdown_connections 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- 172.21.15.129:0/1339404393 wait complete. 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient: build_initial_monmap 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient: monmap: 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:10.687 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 librados: starting msgr at 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 librados: starting objecter 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 Processor -- start 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- start start 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 librados: setting wanted keys 2024-09-18T11:44:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 librados: calling monclient init 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient: init 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient(hunting): picked mon.noname-b con 0x7fb9b806e7e0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient(hunting): picked mon.noname-c con 0x7fb9b809e870 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient(hunting): picked mon.noname-a con 0x7fb9b81322a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b80a0fb0 con 0x7fb9b81322a0 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b806da40 con 0x7fb9b806e7e0 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b811ca00 con 0x7fb9b809e870 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.673+0000 7fb9bde90700 10 monclient(hunting): authenticate will time out at 824.335999s 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bce8e700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb9b806e7e0 0x7fb9b81417f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58192/0 (socket says 172.21.15.129:58192) 2024-09-18T11:44:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bce8e700 1 -- 172.21.15.129:0/2885645509 learned_addr learned my addr 172.21.15.129:0/2885645509 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bce8e700 1 -- 172.21.15.129:0/2885645509 >> v1:172.21.15.129:6789/0 conn(0x7fb9b806e7e0 legacy=0x7fb9b81417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 client.?.objecter ms_handle_connect 0x7fb9b806e7e0 2024-09-18T11:44:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214009050 0 0) 0x7fb9b806da40 con 0x7fb9b806e7e0 2024-09-18T11:44:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): my global_id is 4114 2024-09-18T11:44:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b7fff700 1 -- 172.21.15.129:0/2885645509 >> v1:172.21.15.112:6790/0 conn(0x7fb9b809e870 legacy=0x7fb9b81313a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb99c004f10 con 0x7fb9b806e7e0 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bd68f700 1 -- 172.21.15.129:0/2885645509 >> v1:172.21.15.112:6789/0 conn(0x7fb9b81322a0 legacy=0x7fb9b8149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 client.?.objecter ms_handle_connect 0x7fb9b809e870 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 client.?.objecter ms_handle_connect 0x7fb9b81322a0 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064780552 0 0) 0x7fb9b80a0fb0 con 0x7fb9b81322a0 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): my global_id is 4119 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b806da40 con 0x7fb9b81322a0 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735773509 0 0) 0x7fb9b811ca00 con 0x7fb9b809e870 2024-09-18T11:44:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): my global_id is 4118 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b80a0fb0 con 0x7fb9b809e870 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2331403184 0 0) 0x7fb99c004f10 con 0x7fb9b806e7e0 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9b811ca00 con 0x7fb9b806e7e0 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb9ac001f00 con 0x7fb9b806e7e0 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): dump: 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2192387721 0 0) 0x7fb9b811ca00 con 0x7fb9b806e7e0 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 >> v1:172.21.15.112:6790/0 conn(0x7fb9b809e870 legacy=0x7fb9b81313a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 >> v1:172.21.15.112:6789/0 conn(0x7fb9b81322a0 legacy=0x7fb9b8149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 monclient: found mon.b 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b811e4c0 con 0x7fb9b806e7e0 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient: _finish_auth 0 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 5 monclient: authenticate success, global_id 4114 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 10 monclient: _renew_subs 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b811e7a0 con 0x7fb9b806e7e0 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 10 monclient: _renew_subs 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b8131ab0 con 0x7fb9b806e7e0 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9bde90700 1 librados: init done 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9ac002390 con 0x7fb9b806e7e0 2024-09-18T11:44:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb9ac0027a0 con 0x7fb9b806e7e0 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b6ffd700 4 set_mon_vals no callback set 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 monclient: dump: 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.698 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7fb9ac002c00 con 0x7fb9b806e7e0 2024-09-18T11:44:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (unknown 2107521205 0 0) 0x7fb9ac006410 con 0x7fb9b806e7e0 2024-09-18T11:44:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 10 client.4114.objecter ms_dispatch 0x7fb9b8000fd0 osd_map(3..3 src has 1..3) v4 2024-09-18T11:44:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2024-09-18T11:44:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 3 client.4114.objecter handle_osd_map decoding full epoch 3 2024-09-18T11:44:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.677+0000 7fb9b4ff9700 20 client.4114.objecter dump_active .. 0 homeless 2024-09-18T11:44:10.698 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:10.681+0000 7fb4ba254000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T11:44:10.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.681+0000 7fb9bde90700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.681+0000 7fb9bde90700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.681+0000 7fb9bde90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.681+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9980050b0 con 0x7fb9b806e7e0 2024-09-18T11:44:10.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.685+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7fb9ac005aa0 con 0x7fb9b806e7e0 2024-09-18T11:44:10.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.685+0000 7fb9b4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.685+0000 7fb9b4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:10.744 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:10.737+0000 7fb4ba254000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T11:44:10.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.761+0000 7fb9bde90700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3164789a-58af-4cf7-bd86-260d70dd02c9", "id": 1}] 2024-09-18T11:44:10.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.761+0000 7fb9bde90700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3164789a-58af-4cf7-bd86-260d70dd02c9", "id": 1}] 2024-09-18T11:44:10.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.761+0000 7fb9bde90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.761+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3164789a-58af-4cf7-bd86-260d70dd02c9", "id": 1} v 0) v1 -- 0x7fb9980058e0 con 0x7fb9b806e7e0 2024-09-18T11:44:10.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9b4ff9700 1 -- 172.21.15.129:0/2885645509 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3164789a-58af-4cf7-bd86-260d70dd02c9", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2717525746 0 940483276) 0x7fb9ac006730 con 0x7fb9b806e7e0 2024-09-18T11:44:10.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9b4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3164789a-58af-4cf7-bd86-260d70dd02c9", "id": 1}] 2024-09-18T11:44:10.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9b4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:10.776 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-18T11:44:10.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 10 librados: watch_flush enter 2024-09-18T11:44:10.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 10 librados: watch_flush exit 2024-09-18T11:44:10.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 20 client.4114.objecter shutdown clearing up homeless session... 2024-09-18T11:44:10.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 10 client.4114.objecter successfully canceled tick 2024-09-18T11:44:10.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 10 monclient: shutdown 2024-09-18T11:44:10.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:10.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 >> v1:172.21.15.129:6789/0 conn(0x7fb9b806e7e0 legacy=0x7fb9b81417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 shutdown_connections 2024-09-18T11:44:10.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.769+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 >> 172.21.15.129:0/2885645509 conn(0x7fb9b8068110 msgr2=0x7fb9b806dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:10.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.773+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 shutdown_connections 2024-09-18T11:44:10.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.773+0000 7fb9bde90700 1 -- 172.21.15.129:0/2885645509 wait complete. 2024-09-18T11:44:10.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.773+0000 7fb9bde90700 1 librados: shutdown 2024-09-18T11:44:10.790 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 0223fabc-71f3-4642-8818-d99dbd5ba501 2 2024-09-18T11:44:10.856 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:10.849+0000 7fb4ba254000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient: get_monmap_and_config 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient: build_initial_monmap 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient: monmap: 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:44:10.889781+0000 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:44:10.889781+0000 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:10.892 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:10.892 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:10.892 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 1 Processor -- start 2024-09-18T11:44:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 1 -- start start 2024-09-18T11:44:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient: init 2024-09-18T11:44:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient(hunting): picked mon.noname-b con 0x7f75e80a0500 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient(hunting): picked mon.noname-a con 0x7f75e80a0fe0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient(hunting): picked mon.noname-c con 0x7f75e8097a90 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:10.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80970b0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e8097ed0 con 0x7f75e8097a90 2024-09-18T11:44:10.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80a33d0 con 0x7f75e80a0500 2024-09-18T11:44:10.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:10.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75f02a8700 10 monclient(hunting): authenticate will time out at 824.547097s 2024-09-18T11:44:10.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75ee845700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f75e8097a90 0x7f75e813ad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58194/0 (socket says 172.21.15.129:58194) 2024-09-18T11:44:10.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.885+0000 7f75ee845700 1 -- 172.21.15.129:0/1311560178 learned_addr learned my addr 172.21.15.129:0/1311560178 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:10.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ee845700 1 -- 172.21.15.129:0/1311560178 >> v1:172.21.15.129:6789/0 conn(0x7f75e8097a90 legacy=0x7f75e813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021824263 0 0) 0x7f75e8097ed0 con 0x7f75e8097a90 2024-09-18T11:44:10.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient(hunting): my global_id is 4117 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d4003570 con 0x7f75e8097a90 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ee044700 1 -- 172.21.15.129:0/1311560178 >> v1:172.21.15.112:6790/0 conn(0x7f75e80a0500 legacy=0x7f75e80a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed843700 1 -- 172.21.15.129:0/1311560178 >> v1:172.21.15.112:6789/0 conn(0x7f75e80a0fe0 legacy=0x7f75e8137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2014732480 0 0) 0x7f75d4003570 con 0x7f75e8097a90 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 >> v1:172.21.15.112:6790/0 conn(0x7f75e80a0500 legacy=0x7f75e80a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 >> v1:172.21.15.112:6789/0 conn(0x7f75e80a0fe0 legacy=0x7f75e8137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: _finish_hunting 0 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 monclient: found mon.noname-c 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8096dd0 con 0x7f75e8097a90 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: _finish_auth 0 2024-09-18T11:44:10.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 5 monclient: authenticate success, global_id 4117 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75e0001cc0 con 0x7f75e8097a90 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: dump: 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.899 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75e0001f90 con 0x7f75e8097a90 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 1 -- 172.21.15.129:0/1311560178 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75e0004e10 con 0x7f75e8097a90 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 4 set_mon_vals no callback set 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:10.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75ed042700 10 monclient: dump: 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 10 monclient: shutdown 2024-09-18T11:44:10.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:10.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 1 -- 172.21.15.129:0/1311560178 >> v1:172.21.15.129:6789/0 conn(0x7f75e8097a90 legacy=0x7f75e813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 1 -- 172.21.15.129:0/1311560178 shutdown_connections 2024-09-18T11:44:10.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 1 -- 172.21.15.129:0/1311560178 >> 172.21.15.129:0/1311560178 conn(0x7f75e8067680 msgr2=0x7f75e8067a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:10.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 1 -- 172.21.15.129:0/1311560178 shutdown_connections 2024-09-18T11:44:10.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.889+0000 7f75f02a8700 1 -- 172.21.15.129:0/1311560178 wait complete. 2024-09-18T11:44:10.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient: build_initial_monmap 2024-09-18T11:44:10.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient: monmap: 2024-09-18T11:44:10.906 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 librados: starting msgr at 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 librados: starting objecter 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 Processor -- start 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 -- start start 2024-09-18T11:44:10.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 librados: setting wanted keys 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 librados: calling monclient init 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient: init 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient(hunting): picked mon.noname-a con 0x7f75e8097a90 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient(hunting): picked mon.noname-c con 0x7f75e80a0500 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient(hunting): picked mon.noname-b con 0x7f75e80a0fe0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80a2300 con 0x7f75e8097a90 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80a66f0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e811e480 con 0x7f75e80a0500 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75f02a8700 10 monclient(hunting): authenticate will time out at 824.553881s 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75ee845700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f75e80a0fe0 0x7f75e814a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58200/0 (socket says 172.21.15.129:58200) 2024-09-18T11:44:10.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.893+0000 7f75ee845700 1 -- 172.21.15.129:0/161957274 learned_addr learned my addr 172.21.15.129:0/161957274 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75ee845700 1 -- 172.21.15.129:0/161957274 >> v1:172.21.15.129:6789/0 conn(0x7f75e80a0fe0 legacy=0x7f75e814a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 client.?.objecter ms_handle_connect 0x7f75e80a0fe0 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037702670 0 0) 0x7f75e80a66f0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): my global_id is 4120 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75ee044700 1 -- 172.21.15.129:0/161957274 >> v1:172.21.15.112:6789/0 conn(0x7f75e8097a90 legacy=0x7f75e81322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75cc004f10 con 0x7f75e80a0fe0 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75ed843700 1 -- 172.21.15.129:0/161957274 >> v1:172.21.15.112:6790/0 conn(0x7f75e80a0500 legacy=0x7f75e8146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 client.?.objecter ms_handle_connect 0x7f75e8097a90 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 client.?.objecter ms_handle_connect 0x7f75e80a0500 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37306844 0 0) 0x7f75e80a2300 con 0x7f75e8097a90 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): my global_id is 4125 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e80a66f0 con 0x7f75e8097a90 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862688957 0 0) 0x7f75e811e480 con 0x7f75e80a0500 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): my global_id is 4124 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e80a2300 con 0x7f75e80a0500 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 366641070 0 0) 0x7f75cc004f10 con 0x7f75e80a0fe0 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75e811e480 con 0x7f75e80a0fe0 2024-09-18T11:44:10.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75e0004bb0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.911 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3170670466 0 0) 0x7f75e811e480 con 0x7f75e80a0fe0 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 >> v1:172.21.15.112:6790/0 conn(0x7f75e80a0500 legacy=0x7f75e8146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 >> v1:172.21.15.112:6789/0 conn(0x7f75e8097a90 legacy=0x7f75e81322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 monclient: found mon.b 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e811ddc0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:10.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 5 monclient: authenticate success, global_id 4120 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 10 monclient: _renew_subs 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e811f440 con 0x7f75e80a0fe0 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 10 monclient: _renew_subs 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e8142480 con 0x7f75e80a0fe0 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 1 librados: init done 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75e0004e80 con 0x7f75e80a0fe0 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75e0005560 con 0x7f75e80a0fe0 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:10.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75ec841700 4 set_mon_vals no callback set 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 monclient: dump: 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f75e0005940 con 0x7f75e80a0fe0 2024-09-18T11:44:10.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (unknown 572134526 0 0) 0x7f75e0005bf0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 10 client.4120.objecter ms_dispatch 0x7f75e8000fd0 osd_map(4..4 src has 1..4) v4 2024-09-18T11:44:10.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2024-09-18T11:44:10.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 3 client.4120.objecter handle_osd_map decoding full epoch 4 2024-09-18T11:44:10.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75de7fc700 20 client.4120.objecter dump_active .. 0 homeless 2024-09-18T11:44:10.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.897+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d00050b0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.901+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7f75e0005db0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.901+0000 7f75de7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:10.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.901+0000 7f75de7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.977+0000 7f75f02a8700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0223fabc-71f3-4642-8818-d99dbd5ba501", "id": 2}] 2024-09-18T11:44:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.977+0000 7f75f02a8700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0223fabc-71f3-4642-8818-d99dbd5ba501", "id": 2}] 2024-09-18T11:44:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.977+0000 7f75f02a8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:10.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.977+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0223fabc-71f3-4642-8818-d99dbd5ba501", "id": 2} v 0) v1 -- 0x7f75d00058e0 con 0x7f75e80a0fe0 2024-09-18T11:44:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75de7fc700 1 -- 172.21.15.129:0/161957274 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0223fabc-71f3-4642-8818-d99dbd5ba501", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1526316832 0 216599125) 0x7f75e0006490 con 0x7f75e80a0fe0 2024-09-18T11:44:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75de7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0223fabc-71f3-4642-8818-d99dbd5ba501", "id": 2}] 2024-09-18T11:44:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75de7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:10.991 INFO:teuthology.orchestra.run.smithi129.stdout:2 2024-09-18T11:44:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 10 librados: watch_flush enter 2024-09-18T11:44:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 10 librados: watch_flush exit 2024-09-18T11:44:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 20 client.4120.objecter shutdown clearing up homeless session... 2024-09-18T11:44:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 10 client.4120.objecter successfully canceled tick 2024-09-18T11:44:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 10 monclient: shutdown 2024-09-18T11:44:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 >> v1:172.21.15.129:6789/0 conn(0x7f75e80a0fe0 legacy=0x7f75e814a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 shutdown_connections 2024-09-18T11:44:10.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 >> 172.21.15.129:0/161957274 conn(0x7f75e8067680 msgr2=0x7f75e8135360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:10.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 shutdown_connections 2024-09-18T11:44:10.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 1 -- 172.21.15.129:0/161957274 wait complete. 2024-09-18T11:44:10.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:10.985+0000 7f75f02a8700 1 librados: shutdown 2024-09-18T11:44:11.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 07461389-250c-4e30-8890-4267bddeac4c 3 2024-09-18T11:44:11.083 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:11.077+0000 7fb4ba254000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T11:44:11.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.093+0000 7fc0ae32a700 10 monclient: get_monmap_and_config 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.093+0000 7fc0ae32a700 10 monclient: build_initial_monmap 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.093+0000 7fc0ae32a700 10 monclient: monmap: 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:44:11.099460+0000 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:44:11.099460+0000 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:11.101 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.093+0000 7fc0ae32a700 1 Processor -- start 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.093+0000 7fc0ae32a700 1 -- start start 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.093+0000 7fc0ae32a700 10 monclient: init 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 10 monclient(hunting): picked mon.noname-b con 0x7fc0a809c710 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 10 monclient(hunting): picked mon.noname-c con 0x7fc0a809cae0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 10 monclient(hunting): picked mon.noname-a con 0x7fc0a809b960 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:11.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a80687f0 con 0x7fc0a809b960 2024-09-18T11:44:11.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a809b4d0 con 0x7fc0a809cae0 2024-09-18T11:44:11.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a809e170 con 0x7fc0a809c710 2024-09-18T11:44:11.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:11.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 10 monclient(hunting): authenticate will time out at 824.756929s 2024-09-18T11:44:11.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc09ffff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc0a809cae0 0x7fc0a809da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58204/0 (socket says 172.21.15.129:58204) 2024-09-18T11:44:11.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc09ffff700 1 -- 172.21.15.129:0/1082402681 learned_addr learned my addr 172.21.15.129:0/1082402681 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:11.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc09ffff700 1 -- 172.21.15.129:0/1082402681 >> v1:172.21.15.129:6789/0 conn(0x7fc0a809cae0 legacy=0x7fc0a809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827435917 0 0) 0x7fc0a809b4d0 con 0x7fc0a809cae0 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient(hunting): my global_id is 4123 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc094003570 con 0x7fc0a809cae0 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a7fff700 1 -- 172.21.15.129:0/1082402681 >> v1:172.21.15.112:6790/0 conn(0x7fc0a809c710 legacy=0x7fc0a809d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ac8c7700 1 -- 172.21.15.129:0/1082402681 >> v1:172.21.15.112:6789/0 conn(0x7fc0a809b960 legacy=0x7fc0a813b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864193556 0 0) 0x7fc094003570 con 0x7fc0a809cae0 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 >> v1:172.21.15.112:6790/0 conn(0x7fc0a809c710 legacy=0x7fc0a809d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 >> v1:172.21.15.112:6789/0 conn(0x7fc0a809b960 legacy=0x7fc0a813b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 monclient: found mon.noname-c 2024-09-18T11:44:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a809be00 con 0x7fc0a809cae0 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 5 monclient: authenticate success, global_id 4123 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc090001a00 con 0x7fc0a809cae0 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient: dump: 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0ae32a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc090001be0 con 0x7fc0a809cae0 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.097+0000 7fc0a77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 4 set_mon_vals no callback set 2024-09-18T11:44:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0a77fe700 1 -- 172.21.15.129:0/1082402681 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc090004a90 con 0x7fc0a809cae0 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0a77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0a77fe700 10 monclient: dump: 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.111 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient: shutdown 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- 172.21.15.129:0/1082402681 >> v1:172.21.15.129:6789/0 conn(0x7fc0a809cae0 legacy=0x7fc0a809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- 172.21.15.129:0/1082402681 shutdown_connections 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- 172.21.15.129:0/1082402681 >> 172.21.15.129:0/1082402681 conn(0x7fc0a806a290 msgr2=0x7fc0a80a2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- 172.21.15.129:0/1082402681 shutdown_connections 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- 172.21.15.129:0/1082402681 wait complete. 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient: build_initial_monmap 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient: monmap: 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-09-18T11:44:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-09-18T11:44:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:11.113 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 librados: starting msgr at 2024-09-18T11:44:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 librados: starting objecter 2024-09-18T11:44:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 Processor -- start 2024-09-18T11:44:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- start start 2024-09-18T11:44:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 librados: setting wanted keys 2024-09-18T11:44:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 librados: calling monclient init 2024-09-18T11:44:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient: init 2024-09-18T11:44:11.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:11.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:11.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient(hunting): picked mon.noname-c con 0x7fc0a809b960 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:11.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient(hunting): picked mon.noname-b con 0x7fc0a809c710 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:11.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient(hunting): picked mon.noname-a con 0x7fc0a809cae0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a81219d0 con 0x7fc0a809cae0 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a81216f0 con 0x7fc0a809c710 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a8142520 con 0x7fc0a809b960 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc0ae32a700 10 monclient(hunting): authenticate will time out at 824.762468s 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc09ffff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc0a809c710 0x7fc0a81469a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58212/0 (socket says 172.21.15.129:58212) 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.101+0000 7fc09ffff700 1 -- 172.21.15.129:0/2273221459 learned_addr learned my addr 172.21.15.129:0/2273221459 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc09ffff700 1 -- 172.21.15.129:0/2273221459 >> v1:172.21.15.129:6789/0 conn(0x7fc0a809c710 legacy=0x7fc0a81469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 client.?.objecter ms_handle_connect 0x7fc0a809c710 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225881271 0 0) 0x7fc0a81216f0 con 0x7fc0a809c710 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): my global_id is 4126 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc088004f10 con 0x7fc0a809c710 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 client.?.objecter ms_handle_connect 0x7fc0a809b960 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a7fff700 1 -- 172.21.15.129:0/2273221459 >> v1:172.21.15.112:6790/0 conn(0x7fc0a809b960 legacy=0x7fc0a81342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 client.?.objecter ms_handle_connect 0x7fc0a809cae0 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ac8c7700 1 -- 172.21.15.129:0/2273221459 >> v1:172.21.15.112:6789/0 conn(0x7fc0a809cae0 legacy=0x7fc0a814a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242298124 0 0) 0x7fc088004f10 con 0x7fc0a809c710 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0a81216f0 con 0x7fc0a809c710 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc090004830 con 0x7fc0a809c710 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): dump: 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664987167 0 0) 0x7fc0a81219d0 con 0x7fc0a809cae0 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient(hunting): my global_id is 4131 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc088004f10 con 0x7fc0a809cae0 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 126070811 0 0) 0x7fc0a81216f0 con 0x7fc0a809c710 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 >> v1:172.21.15.112:6790/0 conn(0x7fc0a809b960 legacy=0x7fc0a81342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 >> v1:172.21.15.112:6789/0 conn(0x7fc0a809cae0 legacy=0x7fc0a814a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 monclient: found mon.b 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a811f8e0 con 0x7fc0a809c710 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient: _finish_auth 0 2024-09-18T11:44:11.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 5 monclient: authenticate success, global_id 4126 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 10 monclient: _renew_subs 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a8121410 con 0x7fc0a809c710 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 10 monclient: _renew_subs 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a8136b00 con 0x7fc0a809c710 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 1 librados: init done 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc090004ad0 con 0x7fc0a809c710 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc090005290 con 0x7fc0a809c710 2024-09-18T11:44:11.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a6ffd700 4 set_mon_vals no callback set 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 monclient: dump: 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7fc090005670 con 0x7fc0a809c710 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (unknown 3066208232 0 0) 0x7fc090006700 con 0x7fc0a809c710 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 10 client.4126.objecter ms_dispatch 0x7fc0a8000fd0 osd_map(5..5 src has 1..5) v4 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 3 client.4126.objecter handle_osd_map decoding full epoch 5 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0a4ff9700 20 client.4126.objecter dump_active .. 0 homeless 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.105+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c0050b0 con 0x7fc0a809c710 2024-09-18T11:44:11.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.109+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7fc090005bb0 con 0x7fc0a809c710 2024-09-18T11:44:11.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.109+0000 7fc0a4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.109+0000 7fc0a4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:11.132 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:11.125+0000 7fb4ba254000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T11:44:11.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.189+0000 7fc0ae32a700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "07461389-250c-4e30-8890-4267bddeac4c", "id": 3}] 2024-09-18T11:44:11.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.189+0000 7fc0ae32a700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "07461389-250c-4e30-8890-4267bddeac4c", "id": 3}] 2024-09-18T11:44:11.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.189+0000 7fc0ae32a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.189+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "07461389-250c-4e30-8890-4267bddeac4c", "id": 3} v 0) v1 -- 0x7fc08c0058e0 con 0x7fc0a809c710 2024-09-18T11:44:11.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.193+0000 7fc0a4ff9700 1 -- 172.21.15.129:0/2273221459 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07461389-250c-4e30-8890-4267bddeac4c", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1063892071 0 525046306) 0x7fc090006260 con 0x7fc0a809c710 2024-09-18T11:44:11.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.193+0000 7fc0a4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "07461389-250c-4e30-8890-4267bddeac4c", "id": 3}] 2024-09-18T11:44:11.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.193+0000 7fc0a4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:11.201 INFO:teuthology.orchestra.run.smithi129.stdout:3 2024-09-18T11:44:11.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 10 librados: watch_flush enter 2024-09-18T11:44:11.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 10 librados: watch_flush exit 2024-09-18T11:44:11.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 20 client.4126.objecter shutdown clearing up homeless session... 2024-09-18T11:44:11.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 10 client.4126.objecter successfully canceled tick 2024-09-18T11:44:11.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 10 monclient: shutdown 2024-09-18T11:44:11.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:11.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 >> v1:172.21.15.129:6789/0 conn(0x7fc0a809c710 legacy=0x7fc0a81469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 shutdown_connections 2024-09-18T11:44:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 >> 172.21.15.129:0/2273221459 conn(0x7fc0a806a290 msgr2=0x7fc0a8135410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 shutdown_connections 2024-09-18T11:44:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 1 -- 172.21.15.129:0/2273221459 wait complete. 2024-09-18T11:44:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.197+0000 7fc0ae32a700 1 librados: shutdown 2024-09-18T11:44:11.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new ad3a24d7-0278-4137-a499-e7b7337e1698 4 2024-09-18T11:44:11.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.309+0000 7fd1a6a12700 10 monclient: get_monmap_and_config 2024-09-18T11:44:11.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.309+0000 7fd1a6a12700 10 monclient: build_initial_monmap 2024-09-18T11:44:11.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.309+0000 7fd1a6a12700 10 monclient: monmap: 2024-09-18T11:44:11.317 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:11.317 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:44:11.315458+0000 2024-09-18T11:44:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:44:11.315458+0000 2024-09-18T11:44:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:11.318 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.318 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:11.309+0000 7fb4ba254000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T11:44:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.309+0000 7fd1a6a12700 1 Processor -- start 2024-09-18T11:44:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.309+0000 7fd1a6a12700 1 -- start start 2024-09-18T11:44:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.309+0000 7fd1a6a12700 10 monclient: init 2024-09-18T11:44:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 10 monclient(hunting): picked mon.noname-c con 0x7fd1a009c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 10 monclient(hunting): picked mon.noname-a con 0x7fd1a0067ec0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 10 monclient(hunting): picked mon.noname-b con 0x7fd1a00a1400 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a00a2480 con 0x7fd1a0067ec0 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a0068290 con 0x7fd1a009c670 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a009c010 con 0x7fd1a00a1400 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 10 monclient(hunting): authenticate will time out at 824.972595s 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a5a10700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd1a009c670 0x7fd1a00a05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58214/0 (socket says 172.21.15.129:58214) 2024-09-18T11:44:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a5a10700 1 -- 172.21.15.129:0/4073460607 learned_addr learned my addr 172.21.15.129:0/4073460607 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a5a10700 1 -- 172.21.15.129:0/4073460607 >> v1:172.21.15.129:6789/0 conn(0x7fd1a009c670 legacy=0x7fd1a00a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210751413 0 0) 0x7fd1a0068290 con 0x7fd1a009c670 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient(hunting): my global_id is 4129 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd18c003570 con 0x7fd1a009c670 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a520f700 1 -- 172.21.15.129:0/4073460607 >> v1:172.21.15.112:6789/0 conn(0x7fd1a0067ec0 legacy=0x7fd1a00a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6211700 1 -- 172.21.15.129:0/4073460607 >> v1:172.21.15.112:6790/0 conn(0x7fd1a00a1400 legacy=0x7fd1a013b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2579908531 0 0) 0x7fd18c003570 con 0x7fd1a009c670 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 >> v1:172.21.15.112:6790/0 conn(0x7fd1a00a1400 legacy=0x7fd1a013b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 >> v1:172.21.15.112:6789/0 conn(0x7fd1a0067ec0 legacy=0x7fd1a00a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: _finish_hunting 0 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 monclient: found mon.noname-c 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a0068760 con 0x7fd1a009c670 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: _finish_auth 0 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 5 monclient: authenticate success, global_id 4129 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd19c001940 con 0x7fd1a009c670 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: dump: 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd19c001c10 con 0x7fd1a009c670 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 1 -- 172.21.15.129:0/4073460607 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd19c004a90 con 0x7fd1a009c670 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a6a12700 4 set_mon_vals no callback set 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.313+0000 7fd1a4a0e700 10 monclient: dump: 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient: shutdown 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- 172.21.15.129:0/4073460607 >> v1:172.21.15.129:6789/0 conn(0x7fd1a009c670 legacy=0x7fd1a00a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- 172.21.15.129:0/4073460607 shutdown_connections 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- 172.21.15.129:0/4073460607 >> 172.21.15.129:0/4073460607 conn(0x7fd1a00a3400 msgr2=0x7fd1a009f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- 172.21.15.129:0/4073460607 shutdown_connections 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- 172.21.15.129:0/4073460607 wait complete. 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient: build_initial_monmap 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient: monmap: 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:11.328 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 librados: starting msgr at 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 librados: starting objecter 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 Processor -- start 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- start start 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 librados: setting wanted keys 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 librados: calling monclient init 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient: init 2024-09-18T11:44:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient(hunting): picked mon.noname-a con 0x7fd1a0067ec0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient(hunting): picked mon.noname-b con 0x7fd1a009c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient(hunting): picked mon.noname-c con 0x7fd1a00a1400 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a00a2070 con 0x7fd1a0067ec0 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a0135060 con 0x7fd1a009c670 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a011fe30 con 0x7fd1a00a1400 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a6a12700 10 monclient(hunting): authenticate will time out at 824.978638s 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a520f700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd1a009c670 0x7fd1a0134330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58224/0 (socket says 172.21.15.129:58224) 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.317+0000 7fd1a520f700 1 -- 172.21.15.129:0/1504067986 learned_addr learned my addr 172.21.15.129:0/1504067986 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a520f700 1 -- 172.21.15.129:0/1504067986 >> v1:172.21.15.129:6789/0 conn(0x7fd1a009c670 legacy=0x7fd1a0134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 client.?.objecter ms_handle_connect 0x7fd1a009c670 2024-09-18T11:44:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677856186 0 0) 0x7fd1a0135060 con 0x7fd1a009c670 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): my global_id is 4132 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184004f10 con 0x7fd1a009c670 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 client.?.objecter ms_handle_connect 0x7fd1a0067ec0 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a5a10700 1 -- 172.21.15.129:0/1504067986 >> v1:172.21.15.112:6789/0 conn(0x7fd1a0067ec0 legacy=0x7fd1a0141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 client.?.objecter ms_handle_connect 0x7fd1a00a1400 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6211700 1 -- 172.21.15.129:0/1504067986 >> v1:172.21.15.112:6790/0 conn(0x7fd1a00a1400 legacy=0x7fd1a0149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2230303750 0 0) 0x7fd184004f10 con 0x7fd1a009c670 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1a0135060 con 0x7fd1a009c670 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd190001c60 con 0x7fd1a009c670 2024-09-18T11:44:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): dump: 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.336 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442575005 0 0) 0x7fd1a00a2070 con 0x7fd1a0067ec0 2024-09-18T11:44:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): my global_id is 4137 2024-09-18T11:44:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184004f10 con 0x7fd1a0067ec0 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208658807 0 0) 0x7fd1a011fe30 con 0x7fd1a00a1400 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient(hunting): my global_id is 4136 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a00a2070 con 0x7fd1a00a1400 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1170270739 0 0) 0x7fd1a0135060 con 0x7fd1a009c670 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 >> v1:172.21.15.112:6790/0 conn(0x7fd1a00a1400 legacy=0x7fd1a0149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 >> v1:172.21.15.112:6789/0 conn(0x7fd1a0067ec0 legacy=0x7fd1a0141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 monclient: found mon.b 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a011f920 con 0x7fd1a009c670 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 5 monclient: authenticate success, global_id 4132 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 10 monclient: _renew_subs 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a01204f0 con 0x7fd1a009c670 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 10 monclient: _renew_subs 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a0134a40 con 0x7fd1a009c670 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 1 librados: init done 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd190002340 con 0x7fd1a009c670 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd190002650 con 0x7fd1a009c670 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1977fe700 4 set_mon_vals no callback set 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 monclient: dump: 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.339 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7fd190002af0 con 0x7fd1a009c670 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (unknown 2566488424 0 0) 0x7fd190006400 con 0x7fd1a009c670 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 10 client.4132.objecter ms_dispatch 0x7fd1a0000fd0 osd_map(6..6 src has 1..6) v4 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd195ffb700 20 client.4132.objecter dump_active .. 0 homeless 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.321+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1800050b0 con 0x7fd1a009c670 2024-09-18T11:44:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.325+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7fd190005860 con 0x7fd1a009c670 2024-09-18T11:44:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.325+0000 7fd195ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.325+0000 7fd195ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:11.378 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:11.373+0000 7fb4ba254000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T11:44:11.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.397+0000 7fd1a6a12700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ad3a24d7-0278-4137-a499-e7b7337e1698", "id": 4}] 2024-09-18T11:44:11.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.397+0000 7fd1a6a12700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ad3a24d7-0278-4137-a499-e7b7337e1698", "id": 4}] 2024-09-18T11:44:11.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.397+0000 7fd1a6a12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.397+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ad3a24d7-0278-4137-a499-e7b7337e1698", "id": 4} v 0) v1 -- 0x7fd1800058e0 con 0x7fd1a009c670 2024-09-18T11:44:11.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd195ffb700 1 -- 172.21.15.129:0/1504067986 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ad3a24d7-0278-4137-a499-e7b7337e1698", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4255659279 0 1697012583) 0x7fd190005ef0 con 0x7fd1a009c670 2024-09-18T11:44:11.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd195ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ad3a24d7-0278-4137-a499-e7b7337e1698", "id": 4}] 2024-09-18T11:44:11.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd195ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:11.411 INFO:teuthology.orchestra.run.smithi129.stdout:4 2024-09-18T11:44:11.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 10 librados: watch_flush enter 2024-09-18T11:44:11.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 10 librados: watch_flush exit 2024-09-18T11:44:11.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 20 client.4132.objecter shutdown clearing up homeless session... 2024-09-18T11:44:11.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 10 client.4132.objecter successfully canceled tick 2024-09-18T11:44:11.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 10 monclient: shutdown 2024-09-18T11:44:11.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:11.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 >> v1:172.21.15.129:6789/0 conn(0x7fd1a009c670 legacy=0x7fd1a0134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 shutdown_connections 2024-09-18T11:44:11.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 >> 172.21.15.129:0/1504067986 conn(0x7fd1a00a3400 msgr2=0x7fd1a0135290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:11.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 shutdown_connections 2024-09-18T11:44:11.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 1 -- 172.21.15.129:0/1504067986 wait complete. 2024-09-18T11:44:11.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.405+0000 7fd1a6a12700 1 librados: shutdown 2024-09-18T11:44:11.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 174b7f7f-23bc-45c9-9245-7c3edc4b6235 5 2024-09-18T11:44:11.442 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:11.437+0000 7fb4ba254000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T11:44:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient: get_monmap_and_config 2024-09-18T11:44:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient: build_initial_monmap 2024-09-18T11:44:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient: monmap: 2024-09-18T11:44:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:44:11.518135+0000 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:44:11.518135+0000 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:11.520 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 1 Processor -- start 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 1 -- start start 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient: init 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient(hunting): picked mon.noname-b con 0x7f63f4067680 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient(hunting): picked mon.noname-a con 0x7f63f409db40 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient(hunting): picked mon.noname-c con 0x7f63f406f8c0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f4099cf0 con 0x7f63f409db40 2024-09-18T11:44:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f4067a50 con 0x7f63f406f8c0 2024-09-18T11:44:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f409d8a0 con 0x7f63f4067680 2024-09-18T11:44:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f9154700 10 monclient(hunting): authenticate will time out at 825.175750s 2024-09-18T11:44:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f359e700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f63f406f8c0 0x7f63f41388c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58226/0 (socket says 172.21.15.129:58226) 2024-09-18T11:44:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.513+0000 7f63f359e700 1 -- 172.21.15.129:0/2423115077 learned_addr learned my addr 172.21.15.129:0/2423115077 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f359e700 1 -- 172.21.15.129:0/2423115077 >> v1:172.21.15.129:6789/0 conn(0x7f63f406f8c0 legacy=0x7f63f41388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f2d9d700 1 -- 172.21.15.129:0/2423115077 >> v1:172.21.15.112:6790/0 conn(0x7f63f4067680 legacy=0x7f63f409ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f259c700 1 -- 172.21.15.129:0/2423115077 >> v1:172.21.15.112:6789/0 conn(0x7f63f409db40 legacy=0x7f63f409d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883711099 0 0) 0x7f63f4067a50 con 0x7f63f406f8c0 2024-09-18T11:44:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): my global_id is 4135 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63dc003570 con 0x7f63f406f8c0 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149156921 0 0) 0x7f63f409d8a0 con 0x7f63f4067680 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): my global_id is 4139 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f4067a50 con 0x7f63f4067680 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213482188 0 0) 0x7f63f4099cf0 con 0x7f63f409db40 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient(hunting): my global_id is 4140 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f409d8a0 con 0x7f63f409db40 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801132216 0 0) 0x7f63dc003570 con 0x7f63f406f8c0 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 >> v1:172.21.15.112:6790/0 conn(0x7f63f4067680 legacy=0x7f63f409ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 >> v1:172.21.15.112:6789/0 conn(0x7f63f409db40 legacy=0x7f63f409d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 monclient: found mon.noname-c 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f4099980 con 0x7f63f406f8c0 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: _finish_auth 0 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 5 monclient: authenticate success, global_id 4135 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f63ec001d80 con 0x7f63f406f8c0 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: dump: 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63ec001f60 con 0x7f63f406f8c0 2024-09-18T11:44:11.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 4 set_mon_vals no callback set 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 1 -- 172.21.15.129:0/2423115077 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f63ec004e10 con 0x7f63f406f8c0 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f1d9b700 10 monclient: dump: 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.533 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 10 monclient: shutdown 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 1 -- 172.21.15.129:0/2423115077 >> v1:172.21.15.129:6789/0 conn(0x7f63f406f8c0 legacy=0x7f63f41388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 1 -- 172.21.15.129:0/2423115077 shutdown_connections 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 1 -- 172.21.15.129:0/2423115077 >> 172.21.15.129:0/2423115077 conn(0x7f63f4068b30 msgr2=0x7f63f40a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 1 -- 172.21.15.129:0/2423115077 shutdown_connections 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.517+0000 7f63f9154700 1 -- 172.21.15.129:0/2423115077 wait complete. 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient: build_initial_monmap 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient: monmap: 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:11.534 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 librados: starting msgr at 2024-09-18T11:44:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 librados: starting objecter 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 Processor -- start 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 -- start start 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 librados: setting wanted keys 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 librados: calling monclient init 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient: init 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient(hunting): picked mon.noname-c con 0x7f63f4067680 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient(hunting): picked mon.noname-b con 0x7f63f406f8c0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient(hunting): picked mon.noname-a con 0x7f63f409db40 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f406e620 con 0x7f63f409db40 2024-09-18T11:44:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f4097f30 con 0x7f63f406f8c0 2024-09-18T11:44:11.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f4122a80 con 0x7f63f4067680 2024-09-18T11:44:11.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:11.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f9154700 10 monclient(hunting): authenticate will time out at 825.181670s 2024-09-18T11:44:11.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f259c700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f63f406f8c0 0x7f63f4146ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58238/0 (socket says 172.21.15.129:58238) 2024-09-18T11:44:11.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f259c700 1 -- 172.21.15.129:0/2063768528 learned_addr learned my addr 172.21.15.129:0/2063768528 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:44:11.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63f259c700 1 -- 172.21.15.129:0/2063768528 >> v1:172.21.15.129:6789/0 conn(0x7f63f406f8c0 legacy=0x7f63f4146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63daffd700 10 client.?.objecter ms_handle_connect 0x7f63f406f8c0 2024-09-18T11:44:11.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.521+0000 7f63daffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834050330 0 0) 0x7f63f4097f30 con 0x7f63f406f8c0 2024-09-18T11:44:11.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): my global_id is 4138 2024-09-18T11:44:11.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f2d9d700 1 -- 172.21.15.129:0/2063768528 >> v1:172.21.15.112:6790/0 conn(0x7f63f4067680 legacy=0x7f63f41396c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e0004f10 con 0x7f63f406f8c0 2024-09-18T11:44:11.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f359e700 1 -- 172.21.15.129:0/2063768528 >> v1:172.21.15.112:6789/0 conn(0x7f63f409db40 legacy=0x7f63f414a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 client.?.objecter ms_handle_connect 0x7f63f4067680 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 client.?.objecter ms_handle_connect 0x7f63f409db40 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139818403 0 0) 0x7f63f4122a80 con 0x7f63f4067680 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): my global_id is 4142 2024-09-18T11:44:11.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f4097f30 con 0x7f63f4067680 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936869907 0 0) 0x7f63f406e620 con 0x7f63f409db40 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): my global_id is 4143 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f4122a80 con 0x7f63f409db40 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 347713488 0 0) 0x7f63e0004f10 con 0x7f63f406f8c0 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63f406e620 con 0x7f63f406f8c0 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f63e8001cd0 con 0x7f63f406f8c0 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): dump: 2024-09-18T11:44:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 253636981 0 0) 0x7f63f406e620 con 0x7f63f406f8c0 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 >> v1:172.21.15.112:6790/0 conn(0x7f63f4067680 legacy=0x7f63f41396c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 >> v1:172.21.15.112:6789/0 conn(0x7f63f409db40 legacy=0x7f63f414a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 monclient: found mon.b 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f4139dd0 con 0x7f63f406f8c0 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 5 monclient: authenticate success, global_id 4138 2024-09-18T11:44:11.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 10 monclient: _renew_subs 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f41227a0 con 0x7f63f406f8c0 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 10 monclient: _renew_subs 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f413ee40 con 0x7f63f406f8c0 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 1 librados: init done 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63e80022f0 con 0x7f63f406f8c0 2024-09-18T11:44:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f63e80026c0 con 0x7f63f406f8c0 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f159a700 4 set_mon_vals no callback set 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 monclient: dump: 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-09-18T11:44:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f63e8002aa0 con 0x7f63f406f8c0 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (unknown 2218010542 0 0) 0x7f63e8006420 con 0x7f63f406f8c0 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 10 client.4138.objecter ms_dispatch 0x7f63f4000fd0 osd_map(7..7 src has 1..7) v4 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63daffd700 20 client.4138.objecter dump_active .. 0 homeless 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.525+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d40050b0 con 0x7f63f406f8c0 2024-09-18T11:44:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.529+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7f63e8005870 con 0x7f63f406f8c0 2024-09-18T11:44:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.529+0000 7f63daffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.529+0000 7f63daffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:11.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.605+0000 7f63f9154700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "174b7f7f-23bc-45c9-9245-7c3edc4b6235", "id": 5}] 2024-09-18T11:44:11.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.605+0000 7f63f9154700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "174b7f7f-23bc-45c9-9245-7c3edc4b6235", "id": 5}] 2024-09-18T11:44:11.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.605+0000 7f63f9154700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:11.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.605+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "174b7f7f-23bc-45c9-9245-7c3edc4b6235", "id": 5} v 0) v1 -- 0x7f63d40058e0 con 0x7f63f406f8c0 2024-09-18T11:44:11.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63daffd700 1 -- 172.21.15.129:0/2063768528 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "174b7f7f-23bc-45c9-9245-7c3edc4b6235", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3574100181 0 1988412176) 0x7f63e8005ea0 con 0x7f63f406f8c0 2024-09-18T11:44:11.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63daffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "174b7f7f-23bc-45c9-9245-7c3edc4b6235", "id": 5}] 2024-09-18T11:44:11.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63daffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:11.623 INFO:teuthology.orchestra.run.smithi129.stdout:5 2024-09-18T11:44:11.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 10 librados: watch_flush enter 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 10 librados: watch_flush exit 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 20 client.4138.objecter shutdown clearing up homeless session... 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 10 client.4138.objecter successfully canceled tick 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 10 monclient: shutdown 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 >> v1:172.21.15.129:6789/0 conn(0x7f63f406f8c0 legacy=0x7f63f4146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 shutdown_connections 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 >> 172.21.15.129:0/2063768528 conn(0x7f63f4068b30 msgr2=0x7f63f40a1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 shutdown_connections 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 1 -- 172.21.15.129:0/2063768528 wait complete. 2024-09-18T11:44:11.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-18T11:44:11.617+0000 7f63f9154700 1 librados: shutdown 2024-09-18T11:44:11.637 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T11:44:11.638 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T11:44:11.640 INFO:tasks.ceph.osd.0:Started 2024-09-18T11:44:11.640 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T11:44:11.640 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T11:44:11.642 INFO:tasks.ceph.osd.1:Started 2024-09-18T11:44:11.643 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T11:44:11.643 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T11:44:11.645 INFO:tasks.ceph.osd.2:Started 2024-09-18T11:44:11.645 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T11:44:11.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T11:44:11.647 INFO:tasks.ceph.osd.3:Started 2024-09-18T11:44:11.647 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-18T11:44:11.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-18T11:44:11.649 INFO:tasks.ceph.osd.4:Started 2024-09-18T11:44:11.649 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-18T11:44:11.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-18T11:44:11.652 INFO:tasks.ceph.osd.5:Started 2024-09-18T11:44:11.652 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.747+0000 7f73e9085700 10 monclient: get_monmap_and_config 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.747+0000 7f73e9085700 10 monclient: build_initial_monmap 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.747+0000 7f73e9085700 10 monclient: monmap: 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:11.752325+0000 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:11.752325+0000 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:11.752 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:11.753 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:11.753 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:11.753 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:11.753 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 1 Processor -- start 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 1 -- start start 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 10 monclient: init 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 10 monclient(hunting): picked mon.noname-b con 0x7f73e40a4620 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 10 monclient(hunting): picked mon.noname-c con 0x7f73e409dce0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 10 monclient(hunting): picked mon.noname-a con 0x7f73e4136500 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:11.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e409f300 con 0x7f73e4136500 2024-09-18T11:44:11.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e409fc70 con 0x7f73e409dce0 2024-09-18T11:44:11.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e409feb0 con 0x7f73e40a4620 2024-09-18T11:44:11.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:11.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e9085700 10 monclient(hunting): authenticate will time out at 823.593413s 2024-09-18T11:44:11.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e2d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f73e40a4620 0x7f73e409e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45136/0 (socket says 172.21.15.112:45136) 2024-09-18T11:44:11.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e2d9d700 1 -- 172.21.15.112:0/1305190543 learned_addr learned my addr 172.21.15.112:0/1305190543 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e2d9d700 1 -- 172.21.15.112:0/1305190543 >> v1:172.21.15.112:6790/0 conn(0x7f73e40a4620 legacy=0x7f73e409e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.751+0000 7f73e359e700 1 -- 172.21.15.112:0/1305190543 >> v1:172.21.15.112:6789/0 conn(0x7f73e4136500 legacy=0x7f73e41388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e259c700 1 -- 172.21.15.112:0/1305190543 >> v1:172.21.15.129:6789/0 conn(0x7f73e409dce0 legacy=0x7f73e409ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931550484 0 0) 0x7f73e409feb0 con 0x7f73e40a4620 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient(hunting): my global_id is 4160 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc003570 con 0x7f73e40a4620 2024-09-18T11:44:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47185893 0 0) 0x7f73e409fc70 con 0x7f73e409dce0 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient(hunting): my global_id is 4159 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e409feb0 con 0x7f73e409dce0 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 675021056 0 0) 0x7f73cc003570 con 0x7f73e40a4620 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 >> v1:172.21.15.129:6789/0 conn(0x7f73e409dce0 legacy=0x7f73e409ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 >> v1:172.21.15.112:6789/0 conn(0x7f73e4136500 legacy=0x7f73e41388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 monclient: found mon.noname-b 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e406d4f0 con 0x7f73e40a4620 2024-09-18T11:44:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: _finish_auth 0 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 5 monclient: authenticate success, global_id 4160 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f73d4001940 con 0x7f73e40a4620 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: dump: 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73d4001bd0 con 0x7f73e40a4620 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 1 -- 172.21.15.112:0/1305190543 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f73d4004a90 con 0x7f73e40a4620 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 4 set_mon_vals no callback set 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e1d9b700 10 monclient: dump: 2024-09-18T11:44:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.761 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 10 monclient: shutdown 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 -- 172.21.15.112:0/1305190543 >> v1:172.21.15.112:6790/0 conn(0x7f73e40a4620 legacy=0x7f73e409e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 -- 172.21.15.112:0/1305190543 shutdown_connections 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 -- 172.21.15.112:0/1305190543 >> 172.21.15.112:0/1305190543 conn(0x7f73e4068140 msgr2=0x7f73e40a3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 -- 172.21.15.112:0/1305190543 shutdown_connections 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 -- 172.21.15.112:0/1305190543 wait complete. 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 10 monclient: build_initial_monmap 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 10 monclient: monmap: 2024-09-18T11:44:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 librados: starting msgr at 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 librados: starting objecter 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 Processor -- start 2024-09-18T11:44:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 -- start start 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 librados: setting wanted keys 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 1 librados: calling monclient init 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.755+0000 7f73e9085700 10 monclient: init 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 10 monclient(hunting): picked mon.noname-b con 0x7f73e409dce0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 10 monclient(hunting): picked mon.noname-c con 0x7f73e40a4620 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 10 monclient(hunting): picked mon.noname-a con 0x7f73e4136500 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e406edc0 con 0x7f73e4136500 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e40979c0 con 0x7f73e409dce0 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e4097db0 con 0x7f73e40a4620 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e9085700 10 monclient(hunting): authenticate will time out at 823.600976s 2024-09-18T11:44:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e259c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f73e40a4620 0x7f73e4144770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45148/0 (socket says 172.21.15.112:45148) 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e359e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f73e4136500 0x7f73e4147ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39362/0 (socket says 172.21.15.112:39362) 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e259c700 1 -- 172.21.15.112:0/2605696856 learned_addr learned my addr 172.21.15.112:0/2605696856 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e359e700 1 -- 172.21.15.112:0/2605696856 >> v1:172.21.15.112:6789/0 conn(0x7f73e4136500 legacy=0x7f73e4147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 client.?.objecter ms_handle_connect 0x7f73e4136500 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e259c700 1 -- 172.21.15.112:0/2605696856 >> v1:172.21.15.112:6790/0 conn(0x7f73e40a4620 legacy=0x7f73e4144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 client.?.objecter ms_handle_connect 0x7f73e40a4620 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73e2d9d700 1 -- 172.21.15.112:0/2605696856 >> v1:172.21.15.129:6789/0 conn(0x7f73e409dce0 legacy=0x7f73e409d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 client.?.objecter ms_handle_connect 0x7f73e409dce0 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278833957 0 0) 0x7f73e406edc0 con 0x7f73e4136500 2024-09-18T11:44:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): my global_id is 4158 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d0004f10 con 0x7f73e4136500 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905195118 0 0) 0x7f73e4097db0 con 0x7f73e40a4620 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): my global_id is 4163 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e406edc0 con 0x7f73e40a4620 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492118148 0 0) 0x7f73e40979c0 con 0x7f73e409dce0 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 10 monclient(hunting): my global_id is 4162 2024-09-18T11:44:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.759+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e4097db0 con 0x7f73e409dce0 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929269036 0 0) 0x7f73e406edc0 con 0x7f73e40a4620 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73e40979c0 con 0x7f73e40a4620 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1301328459 0 0) 0x7f73d0004f10 con 0x7f73e4136500 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73e406edc0 con 0x7f73e4136500 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f73d8001d80 con 0x7f73e40a4620 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): dump: 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f73dc001f50 con 0x7f73e4136500 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): dump: 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 440477396 0 0) 0x7f73e4097db0 con 0x7f73e409dce0 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73d0004f10 con 0x7f73e409dce0 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f73d4001720 con 0x7f73e409dce0 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient(hunting): dump: 2024-09-18T11:44:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1279668974 0 0) 0x7f73e40979c0 con 0x7f73e40a4620 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 >> v1:172.21.15.129:6789/0 conn(0x7f73e409dce0 legacy=0x7f73e409d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 >> v1:172.21.15.112:6789/0 conn(0x7f73e4136500 legacy=0x7f73e4147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 monclient: found mon.c 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e4099360 con 0x7f73e40a4620 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 5 monclient: authenticate success, global_id 4163 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 10 monclient: _renew_subs 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e4099640 con 0x7f73e40a4620 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 10 monclient: _renew_subs 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e409d860 con 0x7f73e40a4620 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 1 librados: init done 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73d8002310 con 0x7f73e40a4620 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f73d80026e0 con 0x7f73e40a4620 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e159a700 4 set_mon_vals no callback set 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 monclient: dump: 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:11.778 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f73d8002b40 con 0x7f73e40a4620 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (unknown 996289877 0 0) 0x7f73d8005810 con 0x7f73e40a4620 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 10 client.4163.objecter ms_dispatch 0x7f73e4000fd0 osd_map(8..8 src has 1..8) v4 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 3 client.4163.objecter handle_osd_map got epochs [8,8] > 0 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 3 client.4163.objecter handle_osd_map decoding full epoch 8 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73caffd700 20 client.4163.objecter dump_active .. 0 homeless 2024-09-18T11:44:11.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:11.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.763+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c40050b0 con 0x7f73e40a4620 2024-09-18T11:44:11.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.767+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7f73d8017980 con 0x7f73e40a4620 2024-09-18T11:44:11.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.767+0000 7f73caffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:11.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.767+0000 7f73caffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:11.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:11.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:11.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:11.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c4005360 con 0x7f73e40a4620 2024-09-18T11:44:11.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73caffd700 1 -- 172.21.15.112:0/2605696856 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (unknown 150273500 0 3696387157) 0x7f73d80075c0 con 0x7f73e40a4620 2024-09-18T11:44:11.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73caffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:11.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73caffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:11.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:11.833 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":8,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","created":"2024-09-18T11:44:09.317663+0000","modified":"2024-09-18T11:44:11.615001+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T11:44:11.615001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ecb0abea-5208-453b-86e2-58b64b0634ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3164789a-58af-4cf7-bd86-260d70dd02c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0223fabc-71f3-4642-8818-d99dbd5ba501","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07461389-250c-4e30-8890-4267bddeac4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ad3a24d7-0278-4137-a499-e7b7337e1698","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"174b7f7f-23bc-45c9-9245-7c3edc4b6235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:44:11.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 10 librados: watch_flush enter 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 10 librados: watch_flush exit 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 20 client.4163.objecter shutdown clearing up homeless session... 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 10 client.4163.objecter successfully canceled tick 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 10 monclient: shutdown 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 >> v1:172.21.15.112:6790/0 conn(0x7f73e40a4620 legacy=0x7f73e4144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 shutdown_connections 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 >> 172.21.15.112:0/2605696856 conn(0x7f73e4068140 msgr2=0x7f73e40a19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 shutdown_connections 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 1 -- 172.21.15.112:0/2605696856 wait complete. 2024-09-18T11:44:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:11.831+0000 7f73e9085700 1 librados: shutdown 2024-09-18T11:44:11.845 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-18T11:44:11.845 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T11:44:11.846 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:11.841+0000 7fb4ba254000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T11:44:11.957 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:11.949+0000 7fb4ba254000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T11:44:12.006 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-18T11:44:12.003+0000 7f51d90da540 -1 Falling back to public interface 2024-09-18T11:44:12.247 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:12.241+0000 7fb4ba254000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T11:44:12.262 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-18T11:44:12.259+0000 7fa9ce05d540 -1 Falling back to public interface 2024-09-18T11:44:12.263 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-09-18T11:44:12.257+0000 7f03345d4540 -1 Falling back to public interface 2024-09-18T11:44:12.267 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-18T11:44:12.261+0000 7f7cdf90b540 -1 Falling back to public interface 2024-09-18T11:44:12.280 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-09-18T11:44:12.273+0000 7f5f895e9540 -1 Falling back to public interface 2024-09-18T11:44:12.289 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-18T11:44:12.287+0000 7fd6d4755540 -1 Falling back to public interface 2024-09-18T11:44:12.413 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:12.405+0000 7fb4ba254000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T11:44:12.449 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T11:44:12.479 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:12.473+0000 7fb4ba254000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T11:44:12.533 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:12.525+0000 7fb4ba254000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.547+0000 7fadc359e700 10 monclient: get_monmap_and_config 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.547+0000 7fadc359e700 10 monclient: build_initial_monmap 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.547+0000 7fadc359e700 10 monclient: monmap: 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:44:12.551122+0000 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:44:12.551122+0000 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:12.551 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:12.552 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:12.552 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:12.552 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:12.552 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.547+0000 7fadc359e700 1 Processor -- start 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.547+0000 7fadc359e700 1 -- start start 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.547+0000 7fadc359e700 10 monclient: init 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 10 monclient(hunting): picked mon.noname-c con 0x7fadc40a04f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 10 monclient(hunting): picked mon.noname-a con 0x7fadc40a08c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 10 monclient(hunting): picked mon.noname-b con 0x7fadc4138510 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:12.553 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc4096f10 con 0x7fadc40a08c0 2024-09-18T11:44:12.554 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc40a1f40 con 0x7fadc40a04f0 2024-09-18T11:44:12.554 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc4096c30 con 0x7fadc4138510 2024-09-18T11:44:12.554 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:12.554 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 10 monclient(hunting): authenticate will time out at 824.392179s 2024-09-18T11:44:12.554 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc2d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fadc4138510 0x7fadc413a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45154/0 (socket says 172.21.15.112:45154) 2024-09-18T11:44:12.554 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc2d9d700 1 -- 172.21.15.112:0/1684718869 learned_addr learned my addr 172.21.15.112:0/1684718869 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:12.557 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc1d9b700 1 -- 172.21.15.112:0/1684718869 >> v1:172.21.15.112:6789/0 conn(0x7fadc40a08c0 legacy=0x7fadc40a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:12.557 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc2d9d700 1 -- 172.21.15.112:0/1684718869 >> v1:172.21.15.112:6790/0 conn(0x7fadc4138510 legacy=0x7fadc413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc259c700 1 -- 172.21.15.112:0/1684718869 >> v1:172.21.15.129:6789/0 conn(0x7fadc40a04f0 legacy=0x7fadc40978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448385566 0 0) 0x7fadc4096c30 con 0x7fadc4138510 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): my global_id is 4166 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadac003570 con 0x7fadc4138510 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875427069 0 0) 0x7fadc4096f10 con 0x7fadc40a08c0 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:12.558 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): my global_id is 4161 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4096c30 con 0x7fadc40a08c0 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897842250 0 0) 0x7fadc40a1f40 con 0x7fadc40a04f0 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient(hunting): my global_id is 4165 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4096f10 con 0x7fadc40a04f0 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3451700486 0 0) 0x7fadc4096c30 con 0x7fadc40a08c0 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 >> v1:172.21.15.112:6790/0 conn(0x7fadc4138510 legacy=0x7fadc413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 >> v1:172.21.15.129:6789/0 conn(0x7fadc40a04f0 legacy=0x7fadc40978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: _finish_hunting 0 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 monclient: found mon.noname-a 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:12.559 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc40a13d0 con 0x7fadc40a08c0 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: _finish_auth 0 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 5 monclient: authenticate success, global_id 4161 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fadb4001970 con 0x7fadc40a08c0 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: dump: 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.560 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadb4001c00 con 0x7fadc40a08c0 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:12.561 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:12.562 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc359e700 4 set_mon_vals no callback set 2024-09-18T11:44:12.562 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 1 -- 172.21.15.112:0/1684718869 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fadb40020a0 con 0x7fadc40a08c0 2024-09-18T11:44:12.562 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:12.562 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:12.562 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.551+0000 7fadc159a700 10 monclient: dump: 2024-09-18T11:44:12.562 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:12.562 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient: shutdown 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- 172.21.15.112:0/1684718869 >> v1:172.21.15.112:6789/0 conn(0x7fadc40a08c0 legacy=0x7fadc40a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- 172.21.15.112:0/1684718869 shutdown_connections 2024-09-18T11:44:12.563 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- 172.21.15.112:0/1684718869 >> 172.21.15.112:0/1684718869 conn(0x7fadc40a5b00 msgr2=0x7fadc40a5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- 172.21.15.112:0/1684718869 shutdown_connections 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- 172.21.15.112:0/1684718869 wait complete. 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient: build_initial_monmap 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient: monmap: 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:12.564 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:12.565 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 librados: starting msgr at 2024-09-18T11:44:12.565 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 librados: starting objecter 2024-09-18T11:44:12.565 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 Processor -- start 2024-09-18T11:44:12.565 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- start start 2024-09-18T11:44:12.565 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 librados: setting wanted keys 2024-09-18T11:44:12.565 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 librados: calling monclient init 2024-09-18T11:44:12.565 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient: init 2024-09-18T11:44:12.570 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:12.570 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:12.570 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient(hunting): picked mon.noname-a con 0x7fadc40a04f0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient(hunting): picked mon.noname-c con 0x7fadc40a08c0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient(hunting): picked mon.noname-b con 0x7fadc4138510 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc4137150 con 0x7fadc40a04f0 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc409a940 con 0x7fadc4138510 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc409a6d0 con 0x7fadc40a08c0 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc359e700 10 monclient(hunting): authenticate will time out at 824.399044s 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc259c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fadc40a04f0 0x7fadc409e450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39382/0 (socket says 172.21.15.112:39382) 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.555+0000 7fadc259c700 1 -- 172.21.15.112:0/2283186693 learned_addr learned my addr 172.21.15.112:0/2283186693 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc1d9b700 1 -- 172.21.15.112:0/2283186693 >> v1:172.21.15.112:6790/0 conn(0x7fadc40a08c0 legacy=0x7fadc4146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 client.?.objecter ms_handle_connect 0x7fadc40a08c0 2024-09-18T11:44:12.571 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 client.?.objecter ms_handle_connect 0x7fadc40a04f0 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc259c700 1 -- 172.21.15.112:0/2283186693 >> v1:172.21.15.112:6789/0 conn(0x7fadc40a04f0 legacy=0x7fadc409e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307505092 0 0) 0x7fadc409a6d0 con 0x7fadc40a08c0 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): my global_id is 4169 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc2d9d700 1 -- 172.21.15.112:0/2283186693 >> v1:172.21.15.129:6789/0 conn(0x7fadc4138510 legacy=0x7fadc4149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada4004f10 con 0x7fadc40a08c0 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 client.?.objecter ms_handle_connect 0x7fadc4138510 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873158258 0 0) 0x7fadc4137150 con 0x7fadc40a04f0 2024-09-18T11:44:12.572 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): my global_id is 4164 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc409a6d0 con 0x7fadc40a04f0 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068761177 0 0) 0x7fadc409a940 con 0x7fadc4138510 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): my global_id is 4168 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4137150 con 0x7fadc4138510 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4109055022 0 0) 0x7fada4004f10 con 0x7fadc40a08c0 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc409a940 con 0x7fadc40a08c0 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fadb40020a0 con 0x7fadc40a08c0 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:12.573 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): dump: 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:12.574 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3962886368 0 0) 0x7fadc409a6d0 con 0x7fadc40a04f0 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fada4004f10 con 0x7fadc40a04f0 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fadb8001cf0 con 0x7fadc40a04f0 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): dump: 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:12.575 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571044368 0 0) 0x7fadc4137150 con 0x7fadc4138510 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc409a6d0 con 0x7fadc4138510 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fadbc001ff0 con 0x7fadc4138510 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient(hunting): dump: 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:12.576 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4242235896 0 0) 0x7fada4004f10 con 0x7fadc40a04f0 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 >> v1:172.21.15.112:6790/0 conn(0x7fadc40a08c0 legacy=0x7fadc4146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 >> v1:172.21.15.129:6789/0 conn(0x7fadc4138510 legacy=0x7fadc4149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 monclient: found mon.a 2024-09-18T11:44:12.577 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc409ac20 con 0x7fadc40a04f0 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 5 monclient: authenticate success, global_id 4164 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 10 monclient: _renew_subs 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc40a2150 con 0x7fadc40a04f0 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 10 client.4164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 10 monclient: _renew_subs 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc409ed00 con 0x7fadc40a04f0 2024-09-18T11:44:12.578 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc359e700 1 librados: init done 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadb80021b0 con 0x7fadc40a04f0 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fadb80024c0 con 0x7fadc40a04f0 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadc0d99700 4 set_mon_vals no callback set 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 monclient: dump: 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:12.579 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7fadb8002a50 con 0x7fadc40a04f0 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (unknown 996289877 0 0) 0x7fadb8002d00 con 0x7fadc40a04f0 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 10 client.4164.objecter ms_dispatch 0x7fadc4000fd0 osd_map(8..8 src has 1..8) v4 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 3 client.4164.objecter handle_osd_map got epochs [8,8] > 0 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 3 client.4164.objecter handle_osd_map decoding full epoch 8 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.559+0000 7fadb27fc700 20 client.4164.objecter dump_active .. 0 homeless 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.563+0000 7fadc359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.563+0000 7fadc359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.563+0000 7fadc359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:12.580 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.563+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada80050b0 con 0x7fadc40a04f0 2024-09-18T11:44:12.581 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.563+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (unknown 1092875540 0 1010104650) 0x7fadb8005910 con 0x7fadc40a04f0 2024-09-18T11:44:12.581 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.563+0000 7fadb27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:12.581 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.563+0000 7fadb27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:12.590 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:12.585+0000 7fb4ba254000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T11:44:12.633 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:12.634 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:12.634 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:12.634 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fada8005360 con 0x7fadc40a04f0 2024-09-18T11:44:12.634 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadb27fc700 1 -- 172.21.15.112:0/2283186693 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (unknown 150273500 0 3696387157) 0x7fadb80075c0 con 0x7fadc40a04f0 2024-09-18T11:44:12.634 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadb27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:12.635 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadb27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:12.635 INFO:teuthology.misc.health.smithi112.stdout: 2024-09-18T11:44:12.635 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":8,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","created":"2024-09-18T11:44:09.317663+0000","modified":"2024-09-18T11:44:11.615001+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T11:44:11.615001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ecb0abea-5208-453b-86e2-58b64b0634ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3164789a-58af-4cf7-bd86-260d70dd02c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0223fabc-71f3-4642-8818-d99dbd5ba501","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07461389-250c-4e30-8890-4267bddeac4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ad3a24d7-0278-4137-a499-e7b7337e1698","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"174b7f7f-23bc-45c9-9245-7c3edc4b6235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:44:12.636 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 10 librados: watch_flush enter 2024-09-18T11:44:12.636 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 10 librados: watch_flush exit 2024-09-18T11:44:12.636 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 20 client.4164.objecter shutdown clearing up homeless session... 2024-09-18T11:44:12.636 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 10 client.4164.objecter successfully canceled tick 2024-09-18T11:44:12.636 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 10 monclient: shutdown 2024-09-18T11:44:12.637 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:12.637 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 >> v1:172.21.15.112:6789/0 conn(0x7fadc40a04f0 legacy=0x7fadc409e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:12.637 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 shutdown_connections 2024-09-18T11:44:12.637 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 >> 172.21.15.112:0/2283186693 conn(0x7fadc40a5b00 msgr2=0x7fadc4137730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:12.637 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 shutdown_connections 2024-09-18T11:44:12.637 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 1 -- 172.21.15.112:0/2283186693 wait complete. 2024-09-18T11:44:12.637 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:12.631+0000 7fadc359e700 1 librados: shutdown 2024-09-18T11:44:12.649 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-18T11:44:12.762 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:12.757+0000 7fb4ba254000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T11:44:12.929 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:12.921+0000 7fb4ba254000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T11:44:12.995 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:12.989+0000 7fb4ba254000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T11:44:13.153 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:13.145+0000 7fb4ba254000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T11:44:13.221 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:13.213+0000 7fb4ba254000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T11:44:13.927 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:13.921+0000 7fb4ba254000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T11:44:14.975 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:14.969+0000 7fb4a32cb700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-18T11:44:14.975 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-18T11:44:14.969+0000 7fb4a32cb700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-18T11:44:15.801 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-18T11:44:15.799+0000 7f51d90da540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:44:15.962 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-18T11:44:15.959+0000 7fa9ce05d540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:44:16.003 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-09-18T11:44:15.997+0000 7f5f895e9540 -1 osd.4 0 log_to_monitors true 2024-09-18T11:44:16.091 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-18T11:44:16.085+0000 7f7cdf90b540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:44:16.093 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-09-18T11:44:16.085+0000 7f03345d4540 -1 osd.5 0 log_to_monitors true 2024-09-18T11:44:16.145 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-18T11:44:16.143+0000 7fd6d4755540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:44:17.024 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-18T11:44:17.022+0000 7f51d487a700 -1 osd.1 0 waiting for initial osdmap 2024-09-18T11:44:17.025 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-18T11:44:17.022+0000 7fa9cb022700 -1 osd.0 0 waiting for initial osdmap 2024-09-18T11:44:17.034 INFO:tasks.ceph.osd.0.smithi112.stderr:2024-09-18T11:44:17.030+0000 7fa9c4da5700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:44:17.035 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-18T11:44:17.030+0000 7f51cfe22700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:44:18.028 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-18T11:44:18.026+0000 7fd6cfef5700 -1 osd.2 0 waiting for initial osdmap 2024-09-18T11:44:18.032 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-09-18T11:44:18.025+0000 7f5f84d89700 -1 osd.4 0 waiting for initial osdmap 2024-09-18T11:44:18.032 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-18T11:44:18.025+0000 7f7cdc8d0700 -1 osd.3 0 waiting for initial osdmap 2024-09-18T11:44:18.034 INFO:tasks.ceph.osd.2.smithi112.stderr:2024-09-18T11:44:18.030+0000 7fd6cac9c700 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:44:18.035 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-09-18T11:44:18.029+0000 7f032fd74700 -1 osd.5 0 waiting for initial osdmap 2024-09-18T11:44:18.037 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-09-18T11:44:18.029+0000 7f5f7fb30700 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:44:18.038 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-18T11:44:18.033+0000 7f7cd5e52700 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:44:18.042 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-09-18T11:44:18.037+0000 7f032ab1b700 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:44:19.254 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T11:44:19.356 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.350+0000 7f4926337700 10 monclient: get_monmap_and_config 2024-09-18T11:44:19.356 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.350+0000 7f4926337700 10 monclient: build_initial_monmap 2024-09-18T11:44:19.356 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.350+0000 7f4926337700 10 monclient: monmap: 2024-09-18T11:44:19.356 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-09-18T11:44:19.356 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.357 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:44:19.356326+0000 2024-09-18T11:44:19.357 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:44:19.356326+0000 2024-09-18T11:44:19.357 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:19.357 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.357 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:19.357 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:19.357 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:19.357 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:19.359 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 1 Processor -- start 2024-09-18T11:44:19.359 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 1 -- start start 2024-09-18T11:44:19.359 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 10 monclient: init 2024-09-18T11:44:19.359 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:19.359 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:19.359 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 10 monclient(hunting): picked mon.noname-a con 0x7f492009fe20 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:19.359 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 10 monclient(hunting): picked mon.noname-c con 0x7f4920067d80 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:19.360 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 10 monclient(hunting): picked mon.noname-b con 0x7f49200681b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:19.360 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200a01f0 con 0x7f492009fe20 2024-09-18T11:44:19.360 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4920096c70 con 0x7f4920067d80 2024-09-18T11:44:19.360 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492006fb80 con 0x7f49200681b0 2024-09-18T11:44:19.360 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:19.360 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4926337700 10 monclient(hunting): authenticate will time out at 831.197110s 2024-09-18T11:44:19.360 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4925b36700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f49200681b0 0x7f4920138d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41216/0 (socket says 172.21.15.112:41216) 2024-09-18T11:44:19.361 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4925b36700 1 -- 172.21.15.112:0/4101932371 learned_addr learned my addr 172.21.15.112:0/4101932371 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:19.366 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4925335700 1 -- 172.21.15.112:0/4101932371 >> v1:172.21.15.112:6789/0 conn(0x7f492009fe20 legacy=0x7f4920067670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.366 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4925b36700 1 -- 172.21.15.112:0/4101932371 >> v1:172.21.15.112:6790/0 conn(0x7f49200681b0 legacy=0x7f4920138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.366 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.354+0000 7f4924b34700 1 -- 172.21.15.112:0/4101932371 >> v1:172.21.15.129:6789/0 conn(0x7f4920067d80 legacy=0x7f4920135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.366 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085486532 0 0) 0x7f492006fb80 con 0x7f49200681b0 2024-09-18T11:44:19.366 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.366 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.366 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): my global_id is 4205 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490c003570 con 0x7f49200681b0 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819588509 0 0) 0x7f49200a01f0 con 0x7f492009fe20 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): my global_id is 4203 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492006fb80 con 0x7f492009fe20 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867214204 0 0) 0x7f4920096c70 con 0x7f4920067d80 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.367 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient(hunting): my global_id is 4198 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49200a01f0 con 0x7f4920067d80 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892950203 0 0) 0x7f490c003570 con 0x7f49200681b0 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 >> v1:172.21.15.129:6789/0 conn(0x7f4920067d80 legacy=0x7f4920135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 >> v1:172.21.15.112:6789/0 conn(0x7f492009fe20 legacy=0x7f4920067670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 monclient: found mon.noname-b 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4920095890 con 0x7f49200681b0 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: _finish_auth 0 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:19.368 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4918001cb0 con 0x7f49200681b0 2024-09-18T11:44:19.369 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.369 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:19.369 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: dump: 2024-09-18T11:44:19.369 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:19.369 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.369 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.370 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.370 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.370 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.370 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 5 monclient: authenticate success, global_id 4205 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1883961893 0 0) 0x7f492006fb80 con 0x7f492009fe20 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4910001970 con 0x7f492009fe20 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T11:44:19.371 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4918001e90 con 0x7f49200681b0 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 1 -- 172.21.15.112:0/4101932371 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4918004c70 con 0x7f49200681b0 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 4 set_mon_vals no callback set 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4917fff700 10 monclient: dump: 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:19.372 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 10 monclient: shutdown 2024-09-18T11:44:19.373 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 1 -- 172.21.15.112:0/4101932371 >> v1:172.21.15.112:6790/0 conn(0x7f49200681b0 legacy=0x7f4920138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 1 -- 172.21.15.112:0/4101932371 shutdown_connections 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 1 -- 172.21.15.112:0/4101932371 >> 172.21.15.112:0/4101932371 conn(0x7f49200a1ec0 msgr2=0x7f492009f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 1 -- 172.21.15.112:0/4101932371 shutdown_connections 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 1 -- 172.21.15.112:0/4101932371 wait complete. 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 10 monclient: build_initial_monmap 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.358+0000 7f4926337700 10 monclient: monmap: 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:epoch 0 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:created 0.000000 2024-09-18T11:44:19.374 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 librados: starting msgr at 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 librados: starting objecter 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 Processor -- start 2024-09-18T11:44:19.375 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 -- start start 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 librados: setting wanted keys 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 librados: calling monclient init 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 10 monclient: init 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 10 monclient(hunting): picked mon.noname-b con 0x7f4920067d80 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 10 monclient(hunting): picked mon.noname-a con 0x7f49200681b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 10 monclient(hunting): picked mon.noname-c con 0x7f492009fe20 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492008d800 con 0x7f49200681b0 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492006e9d0 con 0x7f4920067d80 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492008ed40 con 0x7f492009fe20 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:19.376 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4926337700 10 monclient(hunting): authenticate will time out at 831.206011s 2024-09-18T11:44:19.377 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4924b34700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f49200681b0 0x7f492008be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34924/0 (socket says 172.21.15.112:34924) 2024-09-18T11:44:19.377 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.362+0000 7f4924b34700 1 -- 172.21.15.112:0/1073812749 learned_addr learned my addr 172.21.15.112:0/1073812749 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4924b34700 1 -- 172.21.15.112:0/1073812749 >> v1:172.21.15.112:6789/0 conn(0x7f49200681b0 legacy=0x7f492008be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 client.?.objecter ms_handle_connect 0x7f49200681b0 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4925b36700 1 -- 172.21.15.112:0/1073812749 >> v1:172.21.15.112:6790/0 conn(0x7f492009fe20 legacy=0x7f492008c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 DispatchQueue::entry inject delay of 0.002000 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4925335700 1 -- 172.21.15.112:0/1073812749 >> v1:172.21.15.129:6789/0 conn(0x7f4920067d80 legacy=0x7f492008b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 client.?.objecter ms_handle_connect 0x7f492009fe20 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:19.382 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 client.?.objecter ms_handle_connect 0x7f4920067d80 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648290181 0 0) 0x7f492008d800 con 0x7f49200681b0 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): my global_id is 4206 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904004f10 con 0x7f49200681b0 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105813883 0 0) 0x7f492008ed40 con 0x7f492009fe20 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.383 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): my global_id is 4208 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492008d800 con 0x7f492009fe20 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477182342 0 0) 0x7f492006e9d0 con 0x7f4920067d80 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): my global_id is 4201 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492008ed40 con 0x7f4920067d80 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1558540115 0 0) 0x7f492008d800 con 0x7f492009fe20 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f492006e9d0 con 0x7f492009fe20 2024-09-18T11:44:19.384 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4918004bb0 con 0x7f492009fe20 2024-09-18T11:44:19.385 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.385 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:19.385 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): dump: 2024-09-18T11:44:19.385 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:19.385 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.385 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.385 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.386 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.386 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.386 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.386 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.386 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.386 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:19.387 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:19.387 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3060890187 0 0) 0x7f4904004f10 con 0x7f49200681b0 2024-09-18T11:44:19.387 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f492008d800 con 0x7f49200681b0 2024-09-18T11:44:19.387 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f491c001d40 con 0x7f49200681b0 2024-09-18T11:44:19.387 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.387 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:19.388 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient(hunting): dump: 2024-09-18T11:44:19.388 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:19.388 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.388 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.388 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.389 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.389 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.389 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.389 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.389 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.390 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:19.390 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4242397484 0 0) 0x7f492008d800 con 0x7f49200681b0 2024-09-18T11:44:19.390 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 >> v1:172.21.15.112:6790/0 conn(0x7f492009fe20 legacy=0x7f492008c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.390 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 >> v1:172.21.15.129:6789/0 conn(0x7f4920067d80 legacy=0x7f492008b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.390 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient: _finish_hunting 0 2024-09-18T11:44:19.390 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 monclient: found mon.a 2024-09-18T11:44:19.391 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:19.391 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:19.391 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492008fc10 con 0x7f49200681b0 2024-09-18T11:44:19.391 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient: _finish_auth 0 2024-09-18T11:44:19.391 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:19.391 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 5 monclient: authenticate success, global_id 4206 2024-09-18T11:44:19.391 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 10 monclient: _renew_subs 2024-09-18T11:44:19.392 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:19.392 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492008f020 con 0x7f49200681b0 2024-09-18T11:44:19.392 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 10 client.4206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:19.392 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 10 monclient: _renew_subs 2024-09-18T11:44:19.392 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:19.392 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f492008dff0 con 0x7f49200681b0 2024-09-18T11:44:19.393 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4926337700 1 librados: init done 2024-09-18T11:44:19.393 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f491c0022b0 con 0x7f49200681b0 2024-09-18T11:44:19.393 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:19.393 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f491c0026c0 con 0x7f49200681b0 2024-09-18T11:44:19.393 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.394 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:19.394 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f4916ffd700 4 set_mon_vals no callback set 2024-09-18T11:44:19.394 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.366+0000 7f49157fa700 10 monclient: dump: 2024-09-18T11:44:19.394 INFO:teuthology.misc.health.smithi112.stderr:epoch 1 2024-09-18T11:44:19.394 INFO:teuthology.misc.health.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.394 INFO:teuthology.misc.health.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.395 INFO:teuthology.misc.health.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.395 INFO:teuthology.misc.health.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.395 INFO:teuthology.misc.health.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.395 INFO:teuthology.misc.health.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.395 INFO:teuthology.misc.health.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.395 INFO:teuthology.misc.health.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.396 INFO:teuthology.misc.health.smithi112.stderr: 2024-09-18T11:44:19.396 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 4) v1 ==== 42707+0+0 (unknown 2254290883 0 0) 0x7f491c0028a0 con 0x7f49200681b0 2024-09-18T11:44:19.396 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f4926337700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:19.396 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f4926337700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:19.396 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f4926337700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:19.396 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49000050b0 con 0x7f49200681b0 2024-09-18T11:44:19.397 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 2763+0+0 (unknown 2179426774 0 0) 0x7f491c043410 con 0x7f49200681b0 2024-09-18T11:44:19.397 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f49157fa700 10 client.4206.objecter ms_dispatch 0x7f4920000fd0 osd_map(12..12 src has 1..12) v4 2024-09-18T11:44:19.397 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f49157fa700 3 client.4206.objecter handle_osd_map got epochs [12,12] > 0 2024-09-18T11:44:19.397 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f49157fa700 3 client.4206.objecter handle_osd_map decoding full epoch 12 2024-09-18T11:44:19.397 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f49157fa700 20 client.4206.objecter dump_active .. 0 homeless 2024-09-18T11:44:19.397 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.370+0000 7f4925335700 10 monclient: get_auth_request con 0x7f49040380f0 auth_method 0 2024-09-18T11:44:19.398 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.374+0000 7f49157fa700 10 client.4206.objecter ms_handle_connect 0x7f49040380f0 2024-09-18T11:44:19.398 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.378+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f491c014b00 con 0x7f49200681b0 2024-09-18T11:44:19.398 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.378+0000 7f49157fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:19.398 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.378+0000 7f49157fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:19.545 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.542+0000 7f4926337700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:19.545 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.542+0000 7f4926337700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:19.545 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.542+0000 7f4926337700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:19.545 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.542+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4900005360 con 0x7f49200681b0 2024-09-18T11:44:19.546 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.542+0000 7f49157fa700 1 -- 172.21.15.112:0/1073812749 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+6842 (unknown 4250787923 0 3986923756) 0x7f491c014500 con 0x7f49200681b0 2024-09-18T11:44:19.547 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.542+0000 7f49157fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:19.547 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.542+0000 7f49157fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:19.547 INFO:teuthology.misc.health.smithi112.stdout: 2024-09-18T11:44:19.547 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":12,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","created":"2024-09-18T11:44:09.317663+0000","modified":"2024-09-18T11:44:19.034653+0000","last_up_change":"2024-09-18T11:44:19.034653+0000","last_in_change":"2024-09-18T11:44:11.615001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ecb0abea-5208-453b-86e2-58b64b0634ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":20984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":20984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":20984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":20984}]},"public_addr":"172.21.15.112:6804/20984","cluster_addr":"172.21.15.112:6805/20984","heartbeat_back_addr":"172.21.15.112:6807/20984","heartbeat_front_addr":"172.21.15.112:6806/20984","state":["exists","up"]},{"osd":1,"uuid":"3164789a-58af-4cf7-bd86-260d70dd02c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":20985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":20985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":20985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":20985}]},"public_addr":"172.21.15.112:6800/20985","cluster_addr":"172.21.15.112:6801/20985","heartbeat_back_addr":"172.21.15.112:6803/20985","heartbeat_front_addr":"172.21.15.112:6802/20985","state":["exists","up"]},{"osd":2,"uuid":"0223fabc-71f3-4642-8818-d99dbd5ba501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":20986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":20986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":20986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":20986}]},"public_addr":"172.21.15.112:6808/20986","cluster_addr":"172.21.15.112:6809/20986","heartbeat_back_addr":"172.21.15.112:6811/20986","heartbeat_front_addr":"172.21.15.112:6810/20986","state":["exists","up"]},{"osd":3,"uuid":"07461389-250c-4e30-8890-4267bddeac4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6804","nonce":21049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6805","nonce":21049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6807","nonce":21049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6806","nonce":21049}]},"public_addr":"172.21.15.129:6804/21049","cluster_addr":"172.21.15.129:6805/21049","heartbeat_back_addr":"172.21.15.129:6807/21049","heartbeat_front_addr":"172.21.15.129:6806/21049","state":["exists","up"]},{"osd":4,"uuid":"ad3a24d7-0278-4137-a499-e7b7337e1698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6808","nonce":21047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6809","nonce":21047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6811","nonce":21047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6810","nonce":21047}]},"public_addr":"172.21.15.129:6808/21047","cluster_addr":"172.21.15.129:6809/21047","heartbeat_back_addr":"172.21.15.129:6811/21047","heartbeat_front_addr":"172.21.15.129:6810/21047","state":["exists","up"]},{"osd":5,"uuid":"174b7f7f-23bc-45c9-9245-7c3edc4b6235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6800","nonce":21048}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6801","nonce":21048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6803","nonce":21048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6802","nonce":21048}]},"public_addr":"172.21.15.129:6800/21048","cluster_addr":"172.21.15.129:6801/21048","heartbeat_back_addr":"172.21.15.129:6803/21048","heartbeat_front_addr":"172.21.15.129:6802/21048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.965457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.813648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:44:19.549 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 10 librados: watch_flush enter 2024-09-18T11:44:19.549 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 10 librados: watch_flush exit 2024-09-18T11:44:19.549 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 20 client.4206.objecter shutdown clearing up homeless session... 2024-09-18T11:44:19.549 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 10 client.4206.objecter successfully canceled tick 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 >> v1:172.21.15.129:6812/20806 conn(0x7f49040380f0 legacy=0x7f490403a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 10 monclient: shutdown 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 >> v1:172.21.15.112:6789/0 conn(0x7f49200681b0 legacy=0x7f492008be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 shutdown_connections 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 >> 172.21.15.112:0/1073812749 conn(0x7f49200a1ec0 msgr2=0x7f492006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 shutdown_connections 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 1 -- 172.21.15.112:0/1073812749 wait complete. 2024-09-18T11:44:19.550 INFO:teuthology.misc.health.smithi112.stderr:2024-09-18T11:44:19.546+0000 7f4926337700 1 librados: shutdown 2024-09-18T11:44:19.559 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-18T11:44:19.559 INFO:tasks.ceph:Creating RBD pool 2024-09-18T11:44:19.559 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-18T11:44:19.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient: get_monmap_and_config 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient: build_initial_monmap 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient: monmap: 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:19.673049+0000 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:19.673049+0000 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:19.674 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 1 Processor -- start 2024-09-18T11:44:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 1 -- start start 2024-09-18T11:44:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient: init 2024-09-18T11:44:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient(hunting): picked mon.noname-b con 0x7f2594067ef0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient(hunting): picked mon.noname-a con 0x7f259409f880 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient(hunting): picked mon.noname-c con 0x7f25940689d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259409e460 con 0x7f259409f880 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259406e2d0 con 0x7f25940689d0 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259409e9b0 con 0x7f2594067ef0 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259a2a9700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f2594067ef0 0x7f25940682c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41234/0 (socket says 172.21.15.112:41234) 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259b2ab700 10 monclient(hunting): authenticate will time out at 831.515614s 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f259a2a9700 1 -- 172.21.15.112:0/765857736 learned_addr learned my addr 172.21.15.112:0/765857736 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.670+0000 7f2599aa8700 1 --1- 172.21.15.112:0/765857736 >> v1:172.21.15.112:6789/0 conn(0x7f259409f880 0x7f25941354a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34928/0 (socket says 172.21.15.112:34928) 2024-09-18T11:44:19.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f2599aa8700 1 -- 172.21.15.112:0/765857736 >> v1:172.21.15.112:6789/0 conn(0x7f259409f880 legacy=0x7f25941354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f259a2a9700 1 -- 172.21.15.112:0/765857736 >> v1:172.21.15.112:6790/0 conn(0x7f2594067ef0 legacy=0x7f25940682c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f259aaaa700 1 -- 172.21.15.112:0/765857736 >> v1:172.21.15.129:6789/0 conn(0x7f25940689d0 legacy=0x7f2594138bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488093717 0 0) 0x7f259409e460 con 0x7f259409f880 2024-09-18T11:44:19.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): my global_id is 4209 2024-09-18T11:44:19.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2580003570 con 0x7f259409f880 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880425215 0 0) 0x7f259409e9b0 con 0x7f2594067ef0 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): my global_id is 4211 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259409e460 con 0x7f2594067ef0 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279408435 0 0) 0x7f259406e2d0 con 0x7f25940689d0 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient(hunting): my global_id is 4204 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259409e9b0 con 0x7f25940689d0 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1614523077 0 0) 0x7f2580003570 con 0x7f259409f880 2024-09-18T11:44:19.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 >> v1:172.21.15.112:6790/0 conn(0x7f2594067ef0 legacy=0x7f25940682c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 >> v1:172.21.15.129:6789/0 conn(0x7f25940689d0 legacy=0x7f2594138bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f259aaaa700 1 --1- 172.21.15.112:0/765857736 >> v1:172.21.15.129:6789/0 conn(0x7f25940689d0 0x7f2594138bd0 :-1 s=OPENED pgs=54 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f259409e9b0 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: _finish_hunting 0 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 monclient: found mon.noname-a 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259409e6d0 con 0x7f259409f880 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: _finish_auth 0 2024-09-18T11:44:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2584001940 con 0x7f259409f880 2024-09-18T11:44:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f259b2ab700 5 monclient: authenticate success, global_id 4209 2024-09-18T11:44:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f259b2ab700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: dump: 2024-09-18T11:44:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f259b2ab700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547391627 0 0) 0x7f259409e9b0 con 0x7f25940689d0 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f258c001cb0 con 0x7f25940689d0 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2584001bd0 con 0x7f259409f880 2024-09-18T11:44:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f259b2ab700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 1 -- 172.21.15.112:0/765857736 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f25840049d0 con 0x7f259409f880 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f259b2ab700 4 set_mon_vals no callback set 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.674+0000 7f25992a7700 10 monclient: dump: 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient: shutdown 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- 172.21.15.112:0/765857736 >> v1:172.21.15.112:6789/0 conn(0x7f259409f880 legacy=0x7f25941354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- 172.21.15.112:0/765857736 shutdown_connections 2024-09-18T11:44:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- 172.21.15.112:0/765857736 >> 172.21.15.112:0/765857736 conn(0x7f259406a260 msgr2=0x7f25940a2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- 172.21.15.112:0/765857736 shutdown_connections 2024-09-18T11:44:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- 172.21.15.112:0/765857736 wait complete. 2024-09-18T11:44:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient: build_initial_monmap 2024-09-18T11:44:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient: monmap: 2024-09-18T11:44:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 librados: starting msgr at 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 librados: starting objecter 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 Processor -- start 2024-09-18T11:44:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- start start 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 librados: setting wanted keys 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 librados: calling monclient init 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient: init 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient(hunting): picked mon.noname-b con 0x7f2594067ef0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient(hunting): picked mon.noname-a con 0x7f25940689d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient(hunting): picked mon.noname-c con 0x7f259409f880 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259409fd70 con 0x7f25940689d0 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2594096fd0 con 0x7f2594067ef0 2024-09-18T11:44:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259411e770 con 0x7f259409f880 2024-09-18T11:44:19.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:19.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259b2ab700 10 monclient(hunting): authenticate will time out at 831.522889s 2024-09-18T11:44:19.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259aaaa700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f259409f880 0x7f2594147db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41242/0 (socket says 172.21.15.112:41242) 2024-09-18T11:44:19.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.678+0000 7f259aaaa700 1 -- 172.21.15.112:0/116351000 learned_addr learned my addr 172.21.15.112:0/116351000 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:19.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259aaaa700 1 -- 172.21.15.112:0/116351000 >> v1:172.21.15.112:6790/0 conn(0x7f259409f880 legacy=0x7f2594147db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 client.?.objecter ms_handle_connect 0x7f259409f880 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 client.?.objecter ms_handle_connect 0x7f25940689d0 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f2599aa8700 1 -- 172.21.15.112:0/116351000 >> v1:172.21.15.112:6789/0 conn(0x7f25940689d0 legacy=0x7f2594144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 client.?.objecter ms_handle_connect 0x7f2594067ef0 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259a2a9700 1 -- 172.21.15.112:0/116351000 >> v1:172.21.15.129:6789/0 conn(0x7f2594067ef0 legacy=0x7f25941330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967296322 0 0) 0x7f259411e770 con 0x7f259409f880 2024-09-18T11:44:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): my global_id is 4214 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2578004f10 con 0x7f259409f880 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938147329 0 0) 0x7f259409fd70 con 0x7f25940689d0 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): my global_id is 4212 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259411e770 con 0x7f25940689d0 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586245259 0 0) 0x7f2594096fd0 con 0x7f2594067ef0 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): my global_id is 4207 2024-09-18T11:44:19.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259409fd70 con 0x7f2594067ef0 2024-09-18T11:44:19.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 304567129 0 0) 0x7f2578004f10 con 0x7f259409f880 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2594096fd0 con 0x7f259409f880 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f258c002280 con 0x7f259409f880 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.702 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1241296594 0 0) 0x7f259411e770 con 0x7f25940689d0 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2578004f10 con 0x7f25940689d0 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2584004830 con 0x7f25940689d0 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:19.703 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3559573225 0 0) 0x7f259409fd70 con 0x7f2594067ef0 2024-09-18T11:44:19.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f259411e770 con 0x7f2594067ef0 2024-09-18T11:44:19.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2590001720 con 0x7f2594067ef0 2024-09-18T11:44:19.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:19.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:19.705 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:19.705 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2840564033 0 0) 0x7f2594096fd0 con 0x7f259409f880 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 >> v1:172.21.15.129:6789/0 conn(0x7f2594067ef0 legacy=0x7f25941330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 >> v1:172.21.15.112:6789/0 conn(0x7f25940689d0 legacy=0x7f2594144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 monclient: found mon.c 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2594140210 con 0x7f259409f880 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 5 monclient: authenticate success, global_id 4214 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 10 monclient: _renew_subs 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2594120210 con 0x7f259409f880 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 10 monclient: _renew_subs 2024-09-18T11:44:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2594133800 con 0x7f259409f880 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f259b2ab700 1 librados: init done 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f258c002710 con 0x7f259409f880 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f2598aa6700 4 set_mon_vals no callback set 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f258c002b60 con 0x7f259409f880 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.682+0000 7f258a7fc700 10 monclient: dump: 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:19.709 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:19.709 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:19.709 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:19.709 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:19.709 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:19.709 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:19.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f259b2ab700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:19.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f259b2ab700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:19.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f259b2ab700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:19.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c0050b0 con 0x7f259409f880 2024-09-18T11:44:19.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 42707+0+0 (unknown 2254290883 0 0) 0x7f258c002d40 con 0x7f259409f880 2024-09-18T11:44:19.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 2763+0+0 (unknown 2179426774 0 0) 0x7f258c0437f0 con 0x7f259409f880 2024-09-18T11:44:19.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f258a7fc700 10 client.4214.objecter ms_dispatch 0x7f2594000fd0 osd_map(12..12 src has 1..12) v4 2024-09-18T11:44:19.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f258a7fc700 3 client.4214.objecter handle_osd_map got epochs [12,12] > 0 2024-09-18T11:44:19.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f258a7fc700 3 client.4214.objecter handle_osd_map decoding full epoch 12 2024-09-18T11:44:19.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f258a7fc700 20 client.4214.objecter dump_active .. 0 homeless 2024-09-18T11:44:19.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.686+0000 7f259a2a9700 10 monclient: get_auth_request con 0x7f2578038480 auth_method 0 2024-09-18T11:44:19.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.690+0000 7f258a7fc700 10 client.4214.objecter ms_handle_connect 0x7f2578038480 2024-09-18T11:44:19.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.694+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f258c002ff0 con 0x7f259409f880 2024-09-18T11:44:19.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.694+0000 7f258a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:19.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.694+0000 7f258a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:19.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.858+0000 7f259b2ab700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T11:44:19.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.858+0000 7f259b2ab700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T11:44:19.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.858+0000 7f259b2ab700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:19.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:19.858+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f257c0058e0 con 0x7f259409f880 2024-09-18T11:44:20.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.078+0000 7f258a7fc700 1 -- 172.21.15.112:0/116351000 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (unknown 2984082091 0 0) 0x7f258c0148e0 con 0x7f259409f880 2024-09-18T11:44:20.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.078+0000 7f258a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T11:44:20.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.078+0000 7f258a7fc700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-18T11:44:20.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 10 librados: watch_flush enter 2024-09-18T11:44:20.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 10 librados: watch_flush exit 2024-09-18T11:44:20.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 20 client.4214.objecter shutdown clearing up homeless session... 2024-09-18T11:44:20.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 10 client.4214.objecter successfully canceled tick 2024-09-18T11:44:20.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 >> v1:172.21.15.129:6812/20806 conn(0x7f2578038480 legacy=0x7f257803a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:20.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 10 monclient: shutdown 2024-09-18T11:44:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 >> v1:172.21.15.112:6790/0 conn(0x7f259409f880 legacy=0x7f2594147db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 shutdown_connections 2024-09-18T11:44:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 >> 172.21.15.112:0/116351000 conn(0x7f259406a260 msgr2=0x7f259409a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 shutdown_connections 2024-09-18T11:44:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 1 -- 172.21.15.112:0/116351000 wait complete. 2024-09-18T11:44:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.082+0000 7f259b2ab700 1 librados: shutdown 2024-09-18T11:44:20.089 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'rbd' created 2024-09-18T11:44:20.101 DEBUG:teuthology.orchestra.run.smithi112:> rbd --cluster ceph pool init rbd 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.142+0000 7fb6e7e344c0 10 monclient: get_monmap_and_config 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.142+0000 7fb6e7e344c0 10 monclient: build_initial_monmap 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.142+0000 7fb6e7e344c0 10 monclient: monmap: 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:20.147159+0000 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:20.147159+0000 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:20.148 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:20.148 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:20.148 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:20.148 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.142+0000 7fb6e7e344c0 1 Processor -- start 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.142+0000 7fb6e7e344c0 1 -- start start 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.142+0000 7fb6e7e344c0 10 monclient: init 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 10 monclient: _reopen_session rank -1 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 10 monclient(hunting): picked mon.noname-b con 0x564d2bcfa5a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 10 monclient(hunting): picked mon.noname-c con 0x564d2bcf7c50 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 10 monclient(hunting): picked mon.noname-a con 0x564d2bd9c450 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x564d2bc20660 con 0x564d2bd9c450 2024-09-18T11:44:20.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x564d2bbf97f0 con 0x564d2bcf7c50 2024-09-18T11:44:20.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x564d2bc06660 con 0x564d2bcfa5a0 2024-09-18T11:44:20.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 10 monclient(hunting): _renew_subs 2024-09-18T11:44:20.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e7e344c0 10 monclient(hunting): authenticate will time out at 831.988382s 2024-09-18T11:44:20.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e6169700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x564d2bcfa5a0 0x564d2bce5130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41264/0 (socket says 172.21.15.112:41264) 2024-09-18T11:44:20.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e696a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x564d2bd9c450 0x564d2bd9e830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34958/0 (socket says 172.21.15.112:34958) 2024-09-18T11:44:20.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e6169700 1 -- 172.21.15.112:0/3625644675 learned_addr learned my addr 172.21.15.112:0/3625644675 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e6169700 1 -- 172.21.15.112:0/3625644675 >> v1:172.21.15.112:6790/0 conn(0x564d2bcfa5a0 legacy=0x564d2bce5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e696a700 1 -- 172.21.15.112:0/3625644675 >> v1:172.21.15.112:6789/0 conn(0x564d2bd9c450 legacy=0x564d2bd9e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5968700 1 -- 172.21.15.112:0/3625644675 >> v1:172.21.15.129:6789/0 conn(0x564d2bcf7c50 legacy=0x564d2bce5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696267537 0 0) 0x564d2bc20660 con 0x564d2bd9c450 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): my global_id is 4215 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6cc001e10 con 0x564d2bd9c450 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211279205 0 0) 0x564d2bc06660 con 0x564d2bcfa5a0 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): my global_id is 4217 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x564d2bc20660 con 0x564d2bcfa5a0 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720590880 0 0) 0x564d2bbf97f0 con 0x564d2bcf7c50 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 10 monclient(hunting): my global_id is 4210 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x564d2bc06660 con 0x564d2bcf7c50 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.146+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540299458 0 0) 0x7fb6cc001e10 con 0x564d2bd9c450 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 >> v1:172.21.15.112:6790/0 conn(0x564d2bcfa5a0 legacy=0x564d2bce5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 >> v1:172.21.15.129:6789/0 conn(0x564d2bcf7c50 legacy=0x564d2bce5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: _finish_hunting 0 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 monclient: found mon.noname-a 2024-09-18T11:44:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d2bbf9b80 con 0x564d2bd9c450 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: _finish_auth 0 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 5 monclient: authenticate success, global_id 4215 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb6d8001d80 con 0x564d2bd9c450 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: dump: 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:20.155 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410864676 0 0) 0x564d2bc06660 con 0x564d2bcf7c50 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb6d00018b0 con 0x564d2bcf7c50 2024-09-18T11:44:20.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6d8001f60 con 0x564d2bd9c450 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 1 -- 172.21.15.112:0/3625644675 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb6d80035f0 con 0x564d2bd9c450 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 10 monclient: get_monmap_and_config success 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 4 set_mon_vals no callback set 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e5167700 10 monclient: dump: 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.157 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 10 monclient: shutdown 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/3625644675 >> v1:172.21.15.112:6789/0 conn(0x564d2bd9c450 legacy=0x564d2bd9e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/3625644675 shutdown_connections 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/3625644675 >> 172.21.15.112:0/3625644675 conn(0x564d2bce6e70 msgr2=0x564d2bd07c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/3625644675 shutdown_connections 2024-09-18T11:44:20.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/3625644675 wait complete. 2024-09-18T11:44:20.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 10 monclient: build_initial_monmap 2024-09-18T11:44:20.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 10 monclient: monmap: 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 librados: starting msgr at 2024-09-18T11:44:20.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 librados: starting objecter 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 Processor -- start 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 -- start start 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 librados: setting wanted keys 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 1 librados: calling monclient init 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.150+0000 7fb6e7e344c0 10 monclient: init 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 10 monclient: _reopen_session rank -1 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 10 monclient(hunting): picked mon.noname-b con 0x564d2bcfa5a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 10 monclient(hunting): picked mon.noname-a con 0x564d2bd9c450 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 10 monclient(hunting): picked mon.noname-c con 0x564d2bd81960 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x564d2bbf95e0 con 0x564d2bd9c450 2024-09-18T11:44:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x564d2bd02b50 con 0x564d2bcfa5a0 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x564d2bd06b40 con 0x564d2bd81960 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 10 monclient(hunting): _renew_subs 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e7e344c0 10 monclient(hunting): authenticate will time out at 831.996577s 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e5968700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x564d2bd9c450 0x564d2bd9b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34962/0 (socket says 172.21.15.112:34962) 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e5968700 1 -- 172.21.15.112:0/1660518023 learned_addr learned my addr 172.21.15.112:0/1660518023 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e696a700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x564d2bd81960 0x564d2bda9de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41270/0 (socket says 172.21.15.112:41270) 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e6169700 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.129:6789/0 conn(0x564d2bcfa5a0 legacy=0x564d2bda1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e5968700 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.112:6789/0 conn(0x564d2bd9c450 legacy=0x564d2bd9b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 client.?.objecter ms_handle_connect 0x564d2bcfa5a0 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6e696a700 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.112:6790/0 conn(0x564d2bd81960 legacy=0x564d2bda9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 client.?.objecter ms_handle_connect 0x564d2bd81960 2024-09-18T11:44:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 client.?.objecter ms_handle_connect 0x564d2bd9c450 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439945670 0 0) 0x564d2bd02b50 con 0x564d2bcfa5a0 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): my global_id is 4213 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c4002fe0 con 0x564d2bcfa5a0 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468360781 0 0) 0x564d2bbf95e0 con 0x564d2bd9c450 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): my global_id is 4218 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x564d2bd02b50 con 0x564d2bd9c450 2024-09-18T11:44:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565628589 0 0) 0x564d2bd06b40 con 0x564d2bd81960 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 10 monclient(hunting): my global_id is 4220 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.154+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x564d2bbf95e0 con 0x564d2bd81960 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2430755081 0 0) 0x564d2bd02b50 con 0x564d2bd9c450 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x564d2bd06b40 con 0x564d2bd9c450 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb6d0001d80 con 0x564d2bd9c450 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): dump: 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3379080827 0 0) 0x564d2bbf95e0 con 0x564d2bd81960 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x564d2bd02b50 con 0x564d2bd81960 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790813401 0 0) 0x7fb6c4002fe0 con 0x564d2bcfa5a0 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x564d2bbf95e0 con 0x564d2bcfa5a0 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb6d8001cc0 con 0x564d2bd81960 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:20.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): dump: 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb6dc001bf0 con 0x564d2bcfa5a0 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient(hunting): dump: 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 666468562 0 0) 0x564d2bd02b50 con 0x564d2bd81960 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.129:6789/0 conn(0x564d2bcfa5a0 legacy=0x564d2bda1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:20.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.112:6789/0 conn(0x564d2bd9c450 legacy=0x564d2bd9b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 monclient: found mon.c 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d2bd9bf10 con 0x564d2bd81960 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 5 monclient: authenticate success, global_id 4220 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 monclient: _renew_subs 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x564d2bd828c0 con 0x564d2bd81960 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:20.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 monclient: _renew_subs 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x564d2bda2940 con 0x564d2bd81960 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 1 librados: init done 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 librados: wait_for_osdmap waiting 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6d8003b20 con 0x564d2bd81960 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb6d8003e30 con 0x564d2bd81960 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e4966700 4 set_mon_vals no callback set 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 monclient: dump: 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:20.173 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 4) v1 ==== 42707+0+0 (unknown 2254290883 0 0) 0x7fb6d800e670 con 0x564d2bd81960 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(13..13 src has 1..13) v4 ==== 3480+0+0 (unknown 3817943495 0 0) 0x7fb6d8042260 con 0x564d2bd81960 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_map(13..13 src has 1..13) v4 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map got epochs [13,13] > 0 2024-09-18T11:44:20.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map decoding full epoch 13 2024-09-18T11:44:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6d67fc700 20 client.4220.objecter dump_active .. 0 homeless 2024-09-18T11:44:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 librados: wait_for_osdmap done waiting 2024-09-18T11:44:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e6169700 10 monclient: get_auth_request con 0x7fb6c4036810 auth_method 0 2024-09-18T11:44:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T11:44:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T11:44:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.158+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x564d2bdab710 con 0x564d2bd81960 2024-09-18T11:44:20.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:20.162+0000 7fb6d67fc700 10 client.4220.objecter ms_handle_connect 0x7fb6c4036810 2024-09-18T11:44:21.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 141+0+0 (unknown 2078359144 0 0) 0x7fb6d8013330 con 0x564d2bd81960 2024-09-18T11:44:21.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T11:44:21.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-18T11:44:21.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:21.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x564d2bbfad30 con 0x564d2bd81960 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (unknown 1566986600 0 0) 0x7fb6d8013330 con 0x564d2bd81960 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e4966700 10 client.4220.objecter _get_latest_version latest 14, waiting 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e4966700 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e4966700 10 monclient: _renew_subs 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e4966700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e4966700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=14}) v3 -- 0x7fb6c8003b40 con 0x564d2bd81960 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 11 ==== osd_map(14..14 src has 1..14) v4 ==== 1062+0+0 (unknown 661199254 0 0) 0x7fb6d80140c0 con 0x564d2bd81960 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_map(14..14 src has 1..14) v4 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map got epochs [14,14] > 13 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map decoding incremental epoch 14 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6d67fc700 20 client.4220.objecter dump_active .. 0 homeless 2024-09-18T11:44:21.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 10 librados: create oid=rbd_trash nspace= 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 10 client.4220.objecter _op_submit op 0x564d2bdabb80 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 20 client.4220.objecter _calc_target epoch 14 base rbd_trash @1 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 20 client.4220.objecter _calc_target target rbd_trash @1 -> pgid 1.5a749323 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 10 client.4220.objecter _calc_target raw pgid 1.5a749323 -> actual 1.3 acting [1,5] primary 1 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 20 client.4220.objecter _get_session s=0x564d2bc26220 osd=1 3 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 10 client.4220.objecter _op_submit oid rbd_trash '@1' '@1' [create] tid 1 osd.1 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 20 client.4220.objecter get_session s=0x564d2bc26220 osd=1 3 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 15 client.4220.objecter _session_op_assign 1 1 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 15 client.4220.objecter _send_op 1 to 1.3 on osd.1 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6800/20985 -- osd_op(unknown.0.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) v8 -- 0x564d2bdb0080 con 0x564d2bdac580 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 20 client.4220.objecter put_session s=0x564d2bc26220 osd=1 4 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e7e344c0 5 client.4220.objecter 1 in flight 2024-09-18T11:44:21.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.090+0000 7fb6e5968700 10 monclient: get_auth_request con 0x564d2bdac580 auth_method 0 2024-09-18T11:44:21.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.094+0000 7fb6d67fc700 10 client.4220.objecter ms_handle_connect 0x564d2bdac580 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 1 -- 172.21.15.112:0/1660518023 <== osd.1 v1:172.21.15.112:6800/20985 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 ==== 153+0+0 (unknown 1318299302 0 0) 0x7fb6d0002620 con 0x564d2bdac580 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 10 client.4220.objecter in handle_osd_op_reply 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 7 client.4220.objecter handle_osd_op_reply 1 ondisk uv 1 in 1.3 attempt 0 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 10 client.4220.objecter op 0 rval 0 len 0 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 15 client.4220.objecter handle_osd_op_reply completed tid 1 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 15 client.4220.objecter _finish_op 1 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 20 client.4220.objecter put_session s=0x564d2bc26220 osd=1 4 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 15 client.4220.objecter _session_op_remove 1 1 2024-09-18T11:44:21.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e5968700 5 client.4220.objecter 0 in flight 2024-09-18T11:44:21.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.098+0000 7fb6e7e344c0 10 librados: Objecter returned from create r=0 2024-09-18T11:44:21.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 10 client.4220.objecter _op_submit op 0x564d2bdfd5b0 2024-09-18T11:44:21.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 20 client.4220.objecter _calc_target epoch 14 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 20 client.4220.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 10 client.4220.objecter _calc_target raw pgid 1.ac0e573a -> actual 1.2 acting [4,0] primary 4 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 20 client.4220.objecter _get_session s=0x564d2bdfdb20 osd=4 3 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 10 client.4220.objecter _op_submit oid rbd_info '@1' '@1' [call rbd.metadata_list in=33b] tid 2 osd.4 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 20 client.4220.objecter get_session s=0x564d2bdfdb20 osd=4 3 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 15 client.4220.objecter _session_op_assign 4 2 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 15 client.4220.objecter _send_op 2 to 1.2 on osd.4 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.129:6808/21047 -- osd_op(unknown.0.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x564d2be01db0 con 0x564d2bdfe1a0 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 4 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e7e344c0 5 client.4220.objecter 1 in flight 2024-09-18T11:44:21.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.102+0000 7fb6e6169700 10 monclient: get_auth_request con 0x564d2bdfe1a0 auth_method 0 2024-09-18T11:44:21.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:21.106+0000 7fb6d67fc700 10 client.4220.objecter ms_handle_connect 0x564d2bdfe1a0 2024-09-18T11:44:22.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== osd.4 v1:172.21.15.129:6808/21047 1 ==== osd_map(15..15 src has 1..15) v4 ==== 658+0+0 (unknown 1298849756 0 0) 0x7fb6dc0048d0 con 0x564d2bdfe1a0 2024-09-18T11:44:22.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_map(15..15 src has 1..15) v4 2024-09-18T11:44:22.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map got epochs [15,15] > 14 2024-09-18T11:44:22.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map decoding incremental epoch 15 2024-09-18T11:44:22.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 10 client.4220.objecter checking op 2 2024-09-18T11:44:22.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 20 client.4220.objecter _calc_target epoch 15 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:44:22.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 20 client.4220.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 20 client.4220.objecter dump_active .. 0 homeless 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6d67fc700 20 client.4220.objecter 2 1.ac0e573a osd.4 rbd_info [call rbd.metadata_list in=33b] 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 1 -- 172.21.15.112:0/1660518023 <== osd.4 v1:172.21.15.129:6808/21047 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 2769474899 0 0) 0x7fb6dc004c50 con 0x564d2bdfe1a0 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 10 client.4220.objecter in handle_osd_op_reply 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 7 client.4220.objecter handle_osd_op_reply 2 ondisk uv 0 in 1.2 attempt 0 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 10 client.4220.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 10 client.4220.objecter op 0 rval 0 len 0 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 15 client.4220.objecter handle_osd_op_reply completed tid 2 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 15 client.4220.objecter _finish_op 2 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 4 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 15 client.4220.objecter _session_op_remove 4 2 2024-09-18T11:44:22.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e6169700 5 client.4220.objecter 0 in flight 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 10 client.4220.objecter _op_submit op 0x564d2bdfc740 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 20 client.4220.objecter _calc_target epoch 15 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 20 client.4220.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 10 client.4220.objecter _calc_target raw pgid 1.ac0e573a -> actual 1.2 acting [4,0] primary 4 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 20 client.4220.objecter _get_session s=0x564d2bdfdb20 osd=4 3 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 10 client.4220.objecter _op_submit oid rbd_info '@1' '@1' [read 0~0] tid 3 osd.4 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 20 client.4220.objecter get_session s=0x564d2bdfdb20 osd=4 3 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 15 client.4220.objecter _session_op_assign 4 3 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 15 client.4220.objecter _send_op 3 to 1.2 on osd.4 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.129:6808/21047 -- osd_op(unknown.0.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e15) v8 -- 0x564d2bdfcc30 con 0x564d2bdfe1a0 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 4 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.810+0000 7fb6e7e344c0 5 client.4220.objecter 1 in flight 2024-09-18T11:44:22.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 1 -- 172.21.15.112:0/1660518023 <== osd.4 v1:172.21.15.129:6808/21047 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 2755800927 0 0) 0x7fb6dc004c50 con 0x564d2bdfe1a0 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 10 client.4220.objecter in handle_osd_op_reply 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 7 client.4220.objecter handle_osd_op_reply 3 ondisk uv 0 in 1.2 attempt 0 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 10 client.4220.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 10 client.4220.objecter op 0 rval 0 len 0 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 15 client.4220.objecter handle_osd_op_reply completed tid 3 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 15 client.4220.objecter _finish_op 3 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 4 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 15 client.4220.objecter _session_op_remove 4 3 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6e6169700 5 client.4220.objecter 0 in flight 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6d7fff700 10 client.4220.objecter allocate_selfmanaged_snap; pool: 1 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6d7fff700 10 client.4220.objecter pool_op_submit 4 2024-09-18T11:44:22.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:22.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6d7fff700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- pool_op(create unmanaged snap pool 1 tid 4 name v0) v4 -- 0x7fb6bc0030b0 con 0x564d2bd81960 2024-09-18T11:44:22.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:22.814+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 12 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb6d8012ce0 con 0x564d2bd81960 2024-09-18T11:44:23.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.154+0000 7fb6d57fa700 10 monclient: tick 2024-09-18T11:44:23.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.154+0000 7fb6d57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:23.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.154+0000 7fb6d57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:23.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 13 ==== pool_op_reply(tid 4 (0) Success v16) v1 ==== 55+0+0 (unknown 3765343051 0 0) 0x7fb6d8013150 con 0x564d2bd81960 2024-09-18T11:44:23.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 pool_op_reply(tid 4 (0) Success v16) v1 2024-09-18T11:44:23.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 10 client.4220.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v16) v1 2024-09-18T11:44:23.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 10 client.4220.objecter have request 4 at 0x564d2bd82cc0 Op: create unmanaged snap 2024-09-18T11:44:23.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 20 client.4220.objecter waiting for client to reach epoch 16 before calling back 2024-09-18T11:44:23.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 10 monclient: _renew_subs 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=16}) v3 -- 0x564d2bd02b50 con 0x564d2bd81960 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 10 client.4220.objecter done 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 14 ==== osd_map(16..16 src has 1..16) v4 ==== 618+0+0 (unknown 3330872504 0 0) 0x7fb6d803b070 con 0x564d2bd81960 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_map(16..16 src has 1..16) v4 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map got epochs [16,16] > 15 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map decoding incremental epoch 16 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6d67fc700 20 client.4220.objecter dump_active .. 0 homeless 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 20 librados: queue_aio_write 0x564d2bdf88f0 completion 0x564d2bc20280 write_seq 1 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 10 client.4220.objecter _op_submit op 0x7fb6c8006140 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 20 client.4220.objecter _calc_target epoch 16 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:44:23.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 20 client.4220.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 10 client.4220.objecter _calc_target raw pgid 1.ac0e573a -> actual 1.2 acting [4,0] primary 4 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 20 client.4220.objecter _get_session s=0x564d2bdfdb20 osd=4 3 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 10 client.4220.objecter _op_submit oid rbd_info '@1' '@1' [create,write 0~8 in=8b] tid 5 osd.4 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 20 client.4220.objecter get_session s=0x564d2bdfdb20 osd=4 3 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 15 client.4220.objecter _session_op_assign 4 5 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 15 client.4220.objecter _send_op 5 to 1.2 on osd.4 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.129:6808/21047 -- osd_op(unknown.0.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7fb6c8006630 con 0x564d2bdfe1a0 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 4 2024-09-18T11:44:23.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.194+0000 7fb6e4966700 5 client.4220.objecter 1 in flight 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 1 -- 172.21.15.112:0/1660518023 <== osd.4 v1:172.21.15.129:6808/21047 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v16'1 uv1 ondisk = 0) v8 ==== 194+0+0 (unknown 1680118860 0 0) 0x7fb6dc004c50 con 0x564d2bdfe1a0 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_op_reply(5 rbd_info [create,write 0~8] v16'1 uv1 ondisk = 0) v8 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 10 client.4220.objecter in handle_osd_op_reply 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 7 client.4220.objecter handle_osd_op_reply 5 ondisk uv 1 in 1.2 attempt 0 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 10 client.4220.objecter op 0 rval 0 len 0 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 10 client.4220.objecter op 1 rval 0 len 0 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 15 client.4220.objecter handle_osd_op_reply completed tid 5 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 15 client.4220.objecter _finish_op 5 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 4 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 15 client.4220.objecter _session_op_remove 4 5 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 5 client.4220.objecter 0 in flight 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6e6169700 20 librados: complete_aio_write 0x564d2bc20280 2024-09-18T11:44:23.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6d7fff700 10 client.4220.objecter delete_selfmanaged_snap; pool: 1; snap: 2 2024-09-18T11:44:23.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6d7fff700 10 client.4220.objecter pool_op_submit 6 2024-09-18T11:44:23.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:23.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:23.198+0000 7fb6d7fff700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- pool_op(delete unmanaged snap pool 1 tid 6 name v16) v4 -- 0x7fb6bc004550 con 0x564d2bd81960 2024-09-18T11:44:24.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.154+0000 7fb6d57fa700 10 monclient: tick 2024-09-18T11:44:24.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.154+0000 7fb6d57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:24.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 15 ==== pool_op_reply(tid 6 (0) Success v17) v1 ==== 43+0+0 (unknown 3068906746 0 0) 0x7fb6d8013150 con 0x564d2bd81960 2024-09-18T11:44:24.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 pool_op_reply(tid 6 (0) Success v17) v1 2024-09-18T11:44:24.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 10 client.4220.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v17) v1 2024-09-18T11:44:24.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 10 client.4220.objecter have request 6 at 0x7fb6bc0044f0 Op: delete unmanaged snap 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 20 client.4220.objecter waiting for client to reach epoch 17 before calling back 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 10 monclient: _renew_subs 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=17}) v3 -- 0x564d2bd06b40 con 0x564d2bd81960 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 10 client.4220.objecter done 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 1 -- 172.21.15.112:0/1660518023 <== mon.2 v1:172.21.15.112:6790/0 16 ==== osd_map(17..17 src has 1..17) v4 ==== 646+0+0 (unknown 3987336145 0 0) 0x7fb6d803b2c0 con 0x564d2bd81960 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map got epochs [17,17] > 16 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 3 client.4220.objecter handle_osd_map decoding incremental epoch 17 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6d67fc700 20 client.4220.objecter dump_active .. 0 homeless 2024-09-18T11:44:24.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 20 librados: queue_aio_write 0x564d2bdf88f0 completion 0x7fb6bc001d10 write_seq 2 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 10 client.4220.objecter _op_submit op 0x7fb6c8006140 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 20 client.4220.objecter _calc_target epoch 17 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 20 client.4220.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 10 client.4220.objecter _calc_target raw pgid 1.ac0e573a -> actual 1.2 acting [4,0] primary 4 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 20 client.4220.objecter _get_session s=0x564d2bdfdb20 osd=4 3 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 10 client.4220.objecter _op_submit oid rbd_info '@1' '@1' [write 0~19 in=19b] tid 7 osd.4 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 20 client.4220.objecter get_session s=0x564d2bdfdb20 osd=4 3 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 15 client.4220.objecter _session_op_assign 4 7 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 15 client.4220.objecter _send_op 7 to 1.2 on osd.4 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 1 -- 172.21.15.112:0/1660518023 --> v1:172.21.15.129:6808/21047 -- osd_op(unknown.0.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) v8 -- 0x7fb6c8006a00 con 0x564d2bdfe1a0 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 4 2024-09-18T11:44:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.202+0000 7fb6e4966700 5 client.4220.objecter 1 in flight 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 1 -- 172.21.15.112:0/1660518023 <== osd.4 v1:172.21.15.129:6808/21047 5 ==== osd_op_reply(7 rbd_info [write 0~19] v17'2 uv2 ondisk = 0) v8 ==== 152+0+0 (unknown 2451736313 0 0) 0x7fb6dc004c50 con 0x564d2bdfe1a0 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 10 client.4220.objecter ms_dispatch 0x564d2bc72490 osd_op_reply(7 rbd_info [write 0~19] v17'2 uv2 ondisk = 0) v8 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 10 client.4220.objecter in handle_osd_op_reply 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 7 client.4220.objecter handle_osd_op_reply 7 ondisk uv 2 in 1.2 attempt 0 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 10 client.4220.objecter op 0 rval 0 len 0 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 15 client.4220.objecter handle_osd_op_reply completed tid 7 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 15 client.4220.objecter _finish_op 7 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 4 2024-09-18T11:44:24.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 15 client.4220.objecter _session_op_remove 4 7 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 5 client.4220.objecter 0 in flight 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e6169700 20 librados: complete_aio_write 0x7fb6bc001d10 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 10 librados: watch_flush enter 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 10 librados: watch_flush exit 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 10 client.4220.objecter close_session for osd.1 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.112:6800/20985 conn(0x564d2bdac580 legacy=0x564d2bdae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 20 client.4220.objecter put_session s=0x564d2bc26220 osd=1 1 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 10 client.4220.objecter close_session for osd.4 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.129:6808/21047 conn(0x564d2bdfe1a0 legacy=0x564d2be00620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 20 client.4220.objecter put_session s=0x564d2bdfdb20 osd=4 1 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 20 client.4220.objecter shutdown clearing up homeless session... 2024-09-18T11:44:24.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 10 client.4220.objecter successfully canceled tick 2024-09-18T11:44:24.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.129:6812/20806 conn(0x7fb6c4036810 legacy=0x7fb6c4038cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 10 monclient: shutdown 2024-09-18T11:44:24.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:24.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 >> v1:172.21.15.112:6790/0 conn(0x564d2bd81960 legacy=0x564d2bda9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e696a700 1 -- 172.21.15.112:0/1660518023 reap_dead start 2024-09-18T11:44:24.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 shutdown_connections 2024-09-18T11:44:24.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 >> 172.21.15.112:0/1660518023 conn(0x564d2bcf7c50 msgr2=0x564d2bd05b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:24.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 shutdown_connections 2024-09-18T11:44:24.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 -- 172.21.15.112:0/1660518023 wait complete. 2024-09-18T11:44:24.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.214+0000 7fb6e7e344c0 1 librados: shutdown 2024-09-18T11:44:24.223 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-18T11:44:24.223 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T11:44:24.223 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-18T11:44:24.223 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-18T11:44:24.224 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T11:44:24.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient: get_monmap_and_config 2024-09-18T11:44:24.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient: build_initial_monmap 2024-09-18T11:44:24.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient: monmap: 2024-09-18T11:44:24.336 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:24.336 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.336 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:24.336583+0000 2024-09-18T11:44:24.337 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:24.336583+0000 2024-09-18T11:44:24.337 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:24.337 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.337 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:24.337 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:24.337 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:24.337 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 1 Processor -- start 2024-09-18T11:44:24.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 1 -- start start 2024-09-18T11:44:24.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient: init 2024-09-18T11:44:24.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:24.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:24.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient(hunting): picked mon.noname-a con 0x7f8d700a0570 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient(hunting): picked mon.noname-b con 0x7f8d700a1050 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient(hunting): picked mon.noname-c con 0x7f8d70097a50 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70097070 con 0x7f8d700a0570 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70096d90 con 0x7f8d70097a50 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70097e50 con 0x7f8d700a1050 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d76977700 10 monclient(hunting): authenticate will time out at 836.177527s 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d6ffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8d700a0570 0x7f8d700a0940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34974/0 (socket says 172.21.15.112:34974) 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d6f7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8d700a1050 0x7f8d70137780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41282/0 (socket says 172.21.15.112:41282) 2024-09-18T11:44:24.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.334+0000 7f8d6ffff700 1 -- 172.21.15.112:0/745052376 learned_addr learned my addr 172.21.15.112:0/745052376 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:24.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6ffff700 1 -- 172.21.15.112:0/745052376 >> v1:172.21.15.112:6789/0 conn(0x7f8d700a0570 legacy=0x7f8d700a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6f7fe700 1 -- 172.21.15.112:0/745052376 >> v1:172.21.15.112:6790/0 conn(0x7f8d700a1050 legacy=0x7f8d70137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d74f14700 1 -- 172.21.15.112:0/745052376 >> v1:172.21.15.129:6789/0 conn(0x7f8d70097a50 legacy=0x7f8d7013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790095719 0 0) 0x7f8d70097070 con 0x7f8d700a0570 2024-09-18T11:44:24.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): my global_id is 4230 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c003570 con 0x7f8d700a0570 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275779749 0 0) 0x7f8d70097e50 con 0x7f8d700a1050 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): my global_id is 4232 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d70097070 con 0x7f8d700a1050 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80212838 0 0) 0x7f8d70096d90 con 0x7f8d70097a50 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient(hunting): my global_id is 4225 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d70097e50 con 0x7f8d70097a50 2024-09-18T11:44:24.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1664922572 0 0) 0x7f8d70097070 con 0x7f8d700a1050 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 >> v1:172.21.15.129:6789/0 conn(0x7f8d70097a50 legacy=0x7f8d7013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 >> v1:172.21.15.112:6789/0 conn(0x7f8d700a0570 legacy=0x7f8d700a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 monclient: found mon.noname-b 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700a2050 con 0x7f8d700a1050 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: _finish_auth 0 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d58001920 con 0x7f8d700a1050 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 5 monclient: authenticate success, global_id 4232 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:24.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: dump: 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.345 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d58001bb0 con 0x7f8d700a1050 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 1 -- 172.21.15.112:0/745052376 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d580049d0 con 0x7f8d700a1050 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d6effd700 10 monclient: dump: 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.346 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 4 set_mon_vals no callback set 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 10 monclient: shutdown 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 -- 172.21.15.112:0/745052376 >> v1:172.21.15.112:6790/0 conn(0x7f8d700a1050 legacy=0x7f8d70137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 -- 172.21.15.112:0/745052376 shutdown_connections 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 -- 172.21.15.112:0/745052376 >> 172.21.15.112:0/745052376 conn(0x7f8d70067680 msgr2=0x7f8d70067a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 -- 172.21.15.112:0/745052376 shutdown_connections 2024-09-18T11:44:24.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 -- 172.21.15.112:0/745052376 wait complete. 2024-09-18T11:44:24.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 10 monclient: build_initial_monmap 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 10 monclient: monmap: 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 librados: starting msgr at 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 librados: starting objecter 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 Processor -- start 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 -- start start 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 librados: setting wanted keys 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 1 librados: calling monclient init 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.338+0000 7f8d76977700 10 monclient: init 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 10 monclient(hunting): picked mon.noname-c con 0x7f8d70097a50 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 10 monclient(hunting): picked mon.noname-a con 0x7f8d700a0570 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 10 monclient(hunting): picked mon.noname-b con 0x7f8d700a1050 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70142510 con 0x7f8d700a0570 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70122850 con 0x7f8d700a1050 2024-09-18T11:44:24.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70121710 con 0x7f8d70097a50 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d76977700 10 monclient(hunting): authenticate will time out at 836.184642s 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d6f7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8d700a0570 0x7f8d70146960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34984/0 (socket says 172.21.15.112:34984) 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d6f7fe700 1 -- 172.21.15.112:0/661417455 learned_addr learned my addr 172.21.15.112:0/661417455 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d6ffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8d70097a50 0x7f8d70134440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41288/0 (socket says 172.21.15.112:41288) 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d6f7fe700 1 -- 172.21.15.112:0/661417455 >> v1:172.21.15.112:6789/0 conn(0x7f8d700a0570 legacy=0x7f8d70146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d67fff700 10 client.?.objecter ms_handle_connect 0x7f8d700a0570 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d67fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d6ffff700 1 -- 172.21.15.112:0/661417455 >> v1:172.21.15.112:6790/0 conn(0x7f8d70097a50 legacy=0x7f8d70134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d67fff700 10 client.?.objecter ms_handle_connect 0x7f8d70097a50 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d67fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d74f14700 1 -- 172.21.15.112:0/661417455 >> v1:172.21.15.129:6789/0 conn(0x7f8d700a1050 legacy=0x7f8d7014a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.342+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 DispatchQueue::entry inject delay of 0.002000 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 client.?.objecter ms_handle_connect 0x7f8d700a1050 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536744031 0 0) 0x7f8d700a2050 con 0x7f8d700a0570 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): my global_id is 4233 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d50004f10 con 0x7f8d700a0570 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590093741 0 0) 0x7f8d70122850 con 0x7f8d700a1050 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): my global_id is 4228 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d700a2050 con 0x7f8d700a1050 2024-09-18T11:44:24.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263580128 0 0) 0x7f8d70121710 con 0x7f8d70097a50 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): my global_id is 4235 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d70122850 con 0x7f8d70097a50 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2508634251 0 0) 0x7f8d50004f10 con 0x7f8d700a0570 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d70121710 con 0x7f8d700a0570 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d58001d20 con 0x7f8d700a0570 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): dump: 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.357 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2546462041 0 0) 0x7f8d70122850 con 0x7f8d70097a50 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d50004f10 con 0x7f8d70097a50 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d68001f90 con 0x7f8d70097a50 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient(hunting): dump: 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3217983308 0 0) 0x7f8d70121710 con 0x7f8d700a0570 2024-09-18T11:44:24.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 >> v1:172.21.15.112:6790/0 conn(0x7f8d70097a50 legacy=0x7f8d70134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 >> v1:172.21.15.129:6789/0 conn(0x7f8d700a1050 legacy=0x7f8d7014a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 monclient: found mon.a 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70122570 con 0x7f8d700a0570 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient: _finish_auth 0 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 5 monclient: authenticate success, global_id 4233 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 10 monclient: _renew_subs 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d70122b30 con 0x7f8d700a0570 2024-09-18T11:44:24.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 10 monclient: _renew_subs 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7011fee0 con 0x7f8d700a0570 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 1 librados: init done 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d580049c0 con 0x7f8d700a0570 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d76977700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d58005120 con 0x7f8d700a0570 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d6e7fc700 4 set_mon_vals no callback set 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.346+0000 7f8d67fff700 10 monclient: dump: 2024-09-18T11:44:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8d5800f9a0 con 0x7f8d700a0570 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f8d58043830 con 0x7f8d700a0570 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d67fff700 10 client.4233.objecter ms_dispatch 0x7f8d70000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d67fff700 3 client.4233.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:24.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d67fff700 3 client.4233.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:24.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d67fff700 20 client.4233.objecter dump_active .. 0 homeless 2024-09-18T11:44:24.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d76977700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:24.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d76977700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7005f2e0 con 0x7f8d700a0570 2024-09-18T11:44:24.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d6ffff700 10 monclient: get_auth_request con 0x7f8d500385b0 auth_method 0 2024-09-18T11:44:24.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.350+0000 7f8d67fff700 10 client.4233.objecter ms_handle_connect 0x7f8d500385b0 2024-09-18T11:44:24.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.358+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f8d58014bf0 con 0x7f8d700a0570 2024-09-18T11:44:24.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.358+0000 7f8d67fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:24.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.358+0000 7f8d67fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:24.565 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.562+0000 7f8d76977700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-18T11:44:24.565 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.562+0000 7f8d76977700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-18T11:44:24.565 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.562+0000 7f8d76977700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.565 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.562+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f8d700635d0 con 0x7f8d700a0570 2024-09-18T11:44:24.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d67fff700 1 -- 172.21.15.112:0/661417455 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+83348 (unknown 481731318 0 862486408) 0x7f8d580145f0 con 0x7f8d700a0570 2024-09-18T11:44:24.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d67fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-18T11:44:24.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d67fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:24.570 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 10 librados: watch_flush enter 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 10 librados: watch_flush exit 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 20 client.4233.objecter shutdown clearing up homeless session... 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 10 client.4233.objecter successfully canceled tick 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 >> v1:172.21.15.129:6812/20806 conn(0x7f8d500385b0 legacy=0x7f8d5003aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 10 monclient: shutdown 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 >> v1:172.21.15.112:6789/0 conn(0x7f8d700a0570 legacy=0x7f8d70146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 shutdown_connections 2024-09-18T11:44:24.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 >> 172.21.15.112:0/661417455 conn(0x7f8d70067680 msgr2=0x7f8d700a4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:24.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 shutdown_connections 2024-09-18T11:44:24.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 1 -- 172.21.15.112:0/661417455 wait complete. 2024-09-18T11:44:24.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.566+0000 7f8d76977700 1 librados: shutdown 2024-09-18T11:44:24.583 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6812","nonce":20806}]},"active_addr":"172.21.15.129:6812/20806","active_change":"2024-09-18T11:44:13.948251+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":98998374}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":3669125951}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":3385827592}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":3374615903}]}]} 2024-09-18T11:44:24.585 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-18T11:44:24.586 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T11:44:24.586 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:44:24.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient: get_monmap_and_config 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient: build_initial_monmap 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient: monmap: 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:24.696980+0000 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:24.696980+0000 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:24.697 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 1 Processor -- start 2024-09-18T11:44:24.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 1 -- start start 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient: init 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient(hunting): picked mon.noname-a con 0x7f28440a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient(hunting): picked mon.noname-c con 0x7f284412edc0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient(hunting): picked mon.noname-b con 0x7f284412f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28440981a0 con 0x7f28440a2040 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284406c120 con 0x7f284412edc0 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284406eef0 con 0x7f284412f8a0 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284359e700 10 monclient(hunting): authenticate will time out at 836.538452s 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f2842d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f284412f8a0 0x7f2844137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41300/0 (socket says 172.21.15.112:41300) 2024-09-18T11:44:24.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f2842d9d700 1 -- 172.21.15.112:0/1225460164 learned_addr learned my addr 172.21.15.112:0/1225460164 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:24.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.694+0000 7f284259c700 1 --1- 172.21.15.112:0/1225460164 >> v1:172.21.15.112:6789/0 conn(0x7f28440a2040 0x7f2844097a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34992/0 (socket says 172.21.15.112:34992) 2024-09-18T11:44:24.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2842d9d700 1 -- 172.21.15.112:0/1225460164 >> v1:172.21.15.112:6790/0 conn(0x7f284412f8a0 legacy=0x7f2844137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284259c700 1 -- 172.21.15.112:0/1225460164 >> v1:172.21.15.112:6789/0 conn(0x7f28440a2040 legacy=0x7f2844097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f283bfff700 1 -- 172.21.15.112:0/1225460164 >> v1:172.21.15.129:6789/0 conn(0x7f284412edc0 legacy=0x7f284412f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192255428 0 0) 0x7f28440981a0 con 0x7f28440a2040 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): my global_id is 4236 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2830003570 con 0x7f28440a2040 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226450928 0 0) 0x7f284406eef0 con 0x7f284412f8a0 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): my global_id is 4238 2024-09-18T11:44:24.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28440981a0 con 0x7f284412f8a0 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214649389 0 0) 0x7f284406c120 con 0x7f284412edc0 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient(hunting): my global_id is 4231 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284406eef0 con 0x7f284412edc0 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044622911 0 0) 0x7f2830003570 con 0x7f28440a2040 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 >> v1:172.21.15.112:6790/0 conn(0x7f284412f8a0 legacy=0x7f2844137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 >> v1:172.21.15.129:6789/0 conn(0x7f284412edc0 legacy=0x7f284412f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 monclient: found mon.noname-a 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2844096dd0 con 0x7f28440a2040 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: _finish_auth 0 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2834001940 con 0x7f28440a2040 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 5 monclient: authenticate success, global_id 4236 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: dump: 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.705 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2834001bd0 con 0x7f28440a2040 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 1 -- 172.21.15.112:0/1225460164 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f28340049d0 con 0x7f28440a2040 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 4 set_mon_vals no callback set 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:24.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f2841d9b700 10 monclient: dump: 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 10 monclient: shutdown 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:24.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 -- 172.21.15.112:0/1225460164 >> v1:172.21.15.112:6789/0 conn(0x7f28440a2040 legacy=0x7f2844097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 -- 172.21.15.112:0/1225460164 shutdown_connections 2024-09-18T11:44:24.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 -- 172.21.15.112:0/1225460164 >> 172.21.15.112:0/1225460164 conn(0x7f2844067da0 msgr2=0x7f28440a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:24.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 -- 172.21.15.112:0/1225460164 shutdown_connections 2024-09-18T11:44:24.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 -- 172.21.15.112:0/1225460164 wait complete. 2024-09-18T11:44:24.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 10 monclient: build_initial_monmap 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 10 monclient: monmap: 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 librados: starting msgr at 2024-09-18T11:44:24.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 librados: starting objecter 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 Processor -- start 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 -- start start 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 librados: setting wanted keys 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 1 librados: calling monclient init 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.698+0000 7f284359e700 10 monclient: init 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 10 monclient(hunting): picked mon.noname-a con 0x7f28440a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 10 monclient(hunting): picked mon.noname-c con 0x7f284412edc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 10 monclient(hunting): picked mon.noname-b con 0x7f284412f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284406f110 con 0x7f28440a2040 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2844130d40 con 0x7f284412f8a0 2024-09-18T11:44:24.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284411e310 con 0x7f284412edc0 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284359e700 10 monclient(hunting): authenticate will time out at 836.544628s 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283bfff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f284412edc0 0x7f284412e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41304/0 (socket says 172.21.15.112:41304) 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283bfff700 1 -- 172.21.15.112:0/597798574 learned_addr learned my addr 172.21.15.112:0/597798574 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283bfff700 1 -- 172.21.15.112:0/597798574 >> v1:172.21.15.112:6790/0 conn(0x7f284412edc0 legacy=0x7f284412e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 client.?.objecter ms_handle_connect 0x7f284412edc0 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f284259c700 1 -- 172.21.15.112:0/597798574 >> v1:172.21.15.112:6789/0 conn(0x7f28440a2040 legacy=0x7f2844141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 client.?.objecter ms_handle_connect 0x7f28440a2040 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f2842d9d700 1 -- 172.21.15.112:0/597798574 >> v1:172.21.15.129:6789/0 conn(0x7f284412f8a0 legacy=0x7f2844149790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010251823 0 0) 0x7f284411e310 con 0x7f284412edc0 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): my global_id is 4241 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2824004f10 con 0x7f284412edc0 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 client.?.objecter ms_handle_connect 0x7f284412f8a0 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468985191 0 0) 0x7f284406f110 con 0x7f28440a2040 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): my global_id is 4239 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284411e310 con 0x7f28440a2040 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918800478 0 0) 0x7f2844130d40 con 0x7f284412f8a0 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:24.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): my global_id is 4234 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284406f110 con 0x7f284412f8a0 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2731232597 0 0) 0x7f2824004f10 con 0x7f284412edc0 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2844130d40 con 0x7f284412edc0 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f282c001f00 con 0x7f284412edc0 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.717 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1619966293 0 0) 0x7f284411e310 con 0x7f28440a2040 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2824004f10 con 0x7f28440a2040 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2834004830 con 0x7f28440a2040 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.702+0000 7f283a7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.718 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 345853590 0 0) 0x7f2824004f10 con 0x7f28440a2040 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 >> v1:172.21.15.112:6790/0 conn(0x7f284412edc0 legacy=0x7f284412e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 >> v1:172.21.15.129:6789/0 conn(0x7f284412f8a0 legacy=0x7f2844149790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 monclient: found mon.a 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284411de90 con 0x7f28440a2040 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 5 monclient: authenticate success, global_id 4239 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 10 monclient: _renew_subs 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284411f270 con 0x7f28440a2040 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 10 monclient: _renew_subs 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2844142210 con 0x7f28440a2040 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 1 librados: init done 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2834001580 con 0x7f28440a2040 2024-09-18T11:44:24.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2834004f80 con 0x7f28440a2040 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 10 monclient: dump: 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:24.721 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284159a700 4 set_mon_vals no callback set 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f283400f880 con 0x7f28440a2040 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28280050b0 con 0x7f28440a2040 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f2834043790 con 0x7f28440a2040 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 10 client.4239.objecter ms_dispatch 0x7f2844000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 3 client.4239.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 3 client.4239.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283a7fc700 20 client.4239.objecter dump_active .. 0 homeless 2024-09-18T11:44:24.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.706+0000 7f283bfff700 10 monclient: get_auth_request con 0x7f2824038350 auth_method 0 2024-09-18T11:44:24.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.710+0000 7f283a7fc700 10 client.4239.objecter ms_handle_connect 0x7f2824038350 2024-09-18T11:44:24.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.714+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f2834014af0 con 0x7f28440a2040 2024-09-18T11:44:24.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.714+0000 7f283a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:24.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.714+0000 7f283a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:24.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.878+0000 7f284359e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:24.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.878+0000 7f284359e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:24.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.878+0000 7f284359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:24.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.878+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2828005360 con 0x7f28440a2040 2024-09-18T11:44:24.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.878+0000 7f283a7fc700 1 -- 172.21.15.112:0/597798574 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+10108 (unknown 3225238715 0 1823345370) 0x7f28340144f0 con 0x7f28440a2040 2024-09-18T11:44:24.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.878+0000 7f283a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:24.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.878+0000 7f283a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:24.882 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:24.883 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":17,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","created":"2024-09-18T11:44:09.317663+0000","modified":"2024-09-18T11:44:24.194391+0000","last_up_change":"2024-09-18T11:44:19.034653+0000","last_in_change":"2024-09-18T11:44:11.615001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:44:19.901214+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-18T11:44:20.029398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ecb0abea-5208-453b-86e2-58b64b0634ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":20984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":20984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":20984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":20984}]},"public_addr":"172.21.15.112:6804/20984","cluster_addr":"172.21.15.112:6805/20984","heartbeat_back_addr":"172.21.15.112:6807/20984","heartbeat_front_addr":"172.21.15.112:6806/20984","state":["exists","up"]},{"osd":1,"uuid":"3164789a-58af-4cf7-bd86-260d70dd02c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":20985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":20985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":20985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":20985}]},"public_addr":"172.21.15.112:6800/20985","cluster_addr":"172.21.15.112:6801/20985","heartbeat_back_addr":"172.21.15.112:6803/20985","heartbeat_front_addr":"172.21.15.112:6802/20985","state":["exists","up"]},{"osd":2,"uuid":"0223fabc-71f3-4642-8818-d99dbd5ba501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":20986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":20986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":20986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":20986}]},"public_addr":"172.21.15.112:6808/20986","cluster_addr":"172.21.15.112:6809/20986","heartbeat_back_addr":"172.21.15.112:6811/20986","heartbeat_front_addr":"172.21.15.112:6810/20986","state":["exists","up"]},{"osd":3,"uuid":"07461389-250c-4e30-8890-4267bddeac4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6804","nonce":21049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6805","nonce":21049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6807","nonce":21049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6806","nonce":21049}]},"public_addr":"172.21.15.129:6804/21049","cluster_addr":"172.21.15.129:6805/21049","heartbeat_back_addr":"172.21.15.129:6807/21049","heartbeat_front_addr":"172.21.15.129:6806/21049","state":["exists","up"]},{"osd":4,"uuid":"ad3a24d7-0278-4137-a499-e7b7337e1698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6808","nonce":21047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6809","nonce":21047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6811","nonce":21047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6810","nonce":21047}]},"public_addr":"172.21.15.129:6808/21047","cluster_addr":"172.21.15.129:6809/21047","heartbeat_back_addr":"172.21.15.129:6811/21047","heartbeat_front_addr":"172.21.15.129:6810/21047","state":["exists","up"]},{"osd":5,"uuid":"174b7f7f-23bc-45c9-9245-7c3edc4b6235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6800","nonce":21048}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6801","nonce":21048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6803","nonce":21048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6802","nonce":21048}]},"public_addr":"172.21.15.129:6800/21048","cluster_addr":"172.21.15.129:6801/21048","heartbeat_back_addr":"172.21.15.129:6803/21048","heartbeat_front_addr":"172.21.15.129:6802/21048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.965457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.813648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.196234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.055175+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.022483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.092059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:44:24.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.878+0000 7f284359e700 10 librados: watch_flush enter 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 10 librados: watch_flush exit 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 20 client.4239.objecter shutdown clearing up homeless session... 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 10 client.4239.objecter successfully canceled tick 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 >> v1:172.21.15.129:6812/20806 conn(0x7f2824038350 legacy=0x7f282403a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 10 monclient: shutdown 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 >> v1:172.21.15.112:6789/0 conn(0x7f28440a2040 legacy=0x7f2844141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 shutdown_connections 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 >> 172.21.15.112:0/597798574 conn(0x7f2844067da0 msgr2=0x7f284406ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 shutdown_connections 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 1 -- 172.21.15.112:0/597798574 wait complete. 2024-09-18T11:44:24.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.882+0000 7f284359e700 1 librados: shutdown 2024-09-18T11:44:24.895 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T11:44:24.896 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:44:25.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient: monmap: 2024-09-18T11:44:25.002 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.002 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.002 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.002479+0000 2024-09-18T11:44:25.002 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.002479+0000 2024-09-18T11:44:25.003 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.003 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.003 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.003 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.003 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.003 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 1 Processor -- start 2024-09-18T11:44:25.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 1 -- start start 2024-09-18T11:44:25.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient: init 2024-09-18T11:44:25.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient(hunting): picked mon.noname-b con 0x7f21c40a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient(hunting): picked mon.noname-c con 0x7f21c4096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient(hunting): picked mon.noname-a con 0x7f21c413ca20 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c406e910 con 0x7f21c413ca20 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c406e630 con 0x7f21c4096a60 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c40a2270 con 0x7f21c40a1ea0 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c880e700 10 monclient(hunting): authenticate will time out at 836.843293s 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c37fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f21c40a1ea0 0x7f21c406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41308/0 (socket says 172.21.15.112:41308) 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c37fe700 1 -- 172.21.15.112:0/2074512993 learned_addr learned my addr 172.21.15.112:0/2074512993 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:24.998+0000 7f21c3fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f21c413ca20 0x7f21c413ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35020/0 (socket says 172.21.15.112:35020) 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c37fe700 1 -- 172.21.15.112:0/2074512993 >> v1:172.21.15.112:6790/0 conn(0x7f21c40a1ea0 legacy=0x7f21c406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c3fff700 1 -- 172.21.15.112:0/2074512993 >> v1:172.21.15.112:6789/0 conn(0x7f21c413ca20 legacy=0x7f21c413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c2ffd700 1 -- 172.21.15.112:0/2074512993 >> v1:172.21.15.129:6789/0 conn(0x7f21c4096a60 legacy=0x7f21c406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424976876 0 0) 0x7f21c40a2270 con 0x7f21c40a1ea0 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): my global_id is 4244 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ac003570 con 0x7f21c40a1ea0 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568788408 0 0) 0x7f21c406e910 con 0x7f21c413ca20 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): my global_id is 4242 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c40a2270 con 0x7f21c413ca20 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250205023 0 0) 0x7f21c406e630 con 0x7f21c4096a60 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient(hunting): my global_id is 4237 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c406e910 con 0x7f21c4096a60 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91943998 0 0) 0x7f21ac003570 con 0x7f21c40a1ea0 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 >> v1:172.21.15.129:6789/0 conn(0x7f21c4096a60 legacy=0x7f21c406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 >> v1:172.21.15.112:6789/0 conn(0x7f21c413ca20 legacy=0x7f21c413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 monclient: found mon.noname-b 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c406eed0 con 0x7f21c40a1ea0 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f21b8001940 con 0x7f21c40a1ea0 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: dump: 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.011 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 5 monclient: authenticate success, global_id 4244 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21b8001bd0 con 0x7f21c40a1ea0 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 1 -- 172.21.15.112:0/2074512993 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f21b8004a90 con 0x7f21c40a1ea0 2024-09-18T11:44:25.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 4 set_mon_vals no callback set 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c27fc700 10 monclient: dump: 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 10 monclient: shutdown 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 1 -- 172.21.15.112:0/2074512993 >> v1:172.21.15.112:6790/0 conn(0x7f21c40a1ea0 legacy=0x7f21c406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 1 -- 172.21.15.112:0/2074512993 shutdown_connections 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 1 -- 172.21.15.112:0/2074512993 >> 172.21.15.112:0/2074512993 conn(0x7f21c4067da0 msgr2=0x7f21c40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 1 -- 172.21.15.112:0/2074512993 shutdown_connections 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.002+0000 7f21c880e700 1 -- 172.21.15.112:0/2074512993 wait complete. 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient: monmap: 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.014 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 librados: starting msgr at 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 librados: starting objecter 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 Processor -- start 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 -- start start 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 librados: setting wanted keys 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 librados: calling monclient init 2024-09-18T11:44:25.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient: init 2024-09-18T11:44:25.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:25.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient(hunting): picked mon.noname-b con 0x7f21c4096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient(hunting): picked mon.noname-c con 0x7f21c40a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient(hunting): picked mon.noname-a con 0x7f21c413ca20 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c4125390 con 0x7f21c413ca20 2024-09-18T11:44:25.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c406e110 con 0x7f21c4096a60 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c409de80 con 0x7f21c40a1ea0 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c880e700 10 monclient(hunting): authenticate will time out at 836.850490s 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c3fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f21c413ca20 0x7f21c414e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35026/0 (socket says 172.21.15.112:35026) 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c2ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f21c40a1ea0 0x7f21c414b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41324/0 (socket says 172.21.15.112:41324) 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.006+0000 7f21c3fff700 1 -- 172.21.15.112:0/347654350 learned_addr learned my addr 172.21.15.112:0/347654350 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c3fff700 1 -- 172.21.15.112:0/347654350 >> v1:172.21.15.112:6789/0 conn(0x7f21c413ca20 legacy=0x7f21c414e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 client.?.objecter ms_handle_connect 0x7f21c413ca20 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 client.?.objecter ms_handle_connect 0x7f21c40a1ea0 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c2ffd700 1 -- 172.21.15.112:0/347654350 >> v1:172.21.15.112:6790/0 conn(0x7f21c40a1ea0 legacy=0x7f21c414b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c37fe700 1 -- 172.21.15.112:0/347654350 >> v1:172.21.15.129:6789/0 conn(0x7f21c4096a60 legacy=0x7f21c41460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 client.?.objecter ms_handle_connect 0x7f21c4096a60 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144299199 0 0) 0x7f21c4125390 con 0x7f21c413ca20 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): my global_id is 4245 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b0004f10 con 0x7f21c413ca20 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525247412 0 0) 0x7f21c409de80 con 0x7f21c40a1ea0 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): my global_id is 4247 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c4125390 con 0x7f21c40a1ea0 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88162050 0 0) 0x7f21c406e110 con 0x7f21c4096a60 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): my global_id is 4240 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c409de80 con 0x7f21c4096a60 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4054555331 0 0) 0x7f21b0004f10 con 0x7f21c413ca20 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21c406e110 con 0x7f21c413ca20 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f21b4001c30 con 0x7f21c413ca20 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): dump: 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.023 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2076930722 0 0) 0x7f21c409de80 con 0x7f21c4096a60 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b0004f10 con 0x7f21c4096a60 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f21b8004830 con 0x7f21c4096a60 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient(hunting): dump: 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.024 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4039950999 0 0) 0x7f21c406e110 con 0x7f21c413ca20 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 >> v1:172.21.15.112:6790/0 conn(0x7f21c40a1ea0 legacy=0x7f21c414b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 >> v1:172.21.15.129:6789/0 conn(0x7f21c4096a60 legacy=0x7f21c41460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 monclient: found mon.a 2024-09-18T11:44:25.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c4124e10 con 0x7f21c413ca20 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 5 monclient: authenticate success, global_id 4245 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 10 monclient: _renew_subs 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c41259e0 con 0x7f21c413ca20 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 10 monclient: _renew_subs 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c4124420 con 0x7f21c413ca20 2024-09-18T11:44:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 1 librados: init done 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21b4004ae0 con 0x7f21c413ca20 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f21b4004e30 con 0x7f21c413ca20 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c1ffb700 4 set_mon_vals no callback set 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 10 monclient: dump: 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.027 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a40050b0 con 0x7f21c413ca20 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.010+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f21b4004f90 con 0x7f21c413ca20 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.014+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f21b4043680 con 0x7f21c413ca20 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.014+0000 7f21ab7fe700 10 client.4245.objecter ms_dispatch 0x7f21c4000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.014+0000 7f21ab7fe700 3 client.4245.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.014+0000 7f21ab7fe700 3 client.4245.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.014+0000 7f21ab7fe700 20 client.4245.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.014+0000 7f21c37fe700 10 monclient: get_auth_request con 0x7f21b00383e0 auth_method 0 2024-09-18T11:44:25.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.014+0000 7f21ab7fe700 10 client.4245.objecter ms_handle_connect 0x7f21b00383e0 2024-09-18T11:44:25.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.022+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f21b4014920 con 0x7f21c413ca20 2024-09-18T11:44:25.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.022+0000 7f21ab7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.022+0000 7f21ab7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:25.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.186+0000 7f21c880e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.186+0000 7f21c880e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.186+0000 7f21c880e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.186+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21a4005360 con 0x7f21c413ca20 2024-09-18T11:44:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.186+0000 7f21ab7fe700 1 -- 172.21.15.112:0/347654350 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+10108 (unknown 3225238715 0 1823345370) 0x7f21b4014320 con 0x7f21c413ca20 2024-09-18T11:44:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.186+0000 7f21ab7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.186+0000 7f21ab7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:25.192 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:25.192 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":17,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","created":"2024-09-18T11:44:09.317663+0000","modified":"2024-09-18T11:44:24.194391+0000","last_up_change":"2024-09-18T11:44:19.034653+0000","last_in_change":"2024-09-18T11:44:11.615001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:44:19.901214+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-18T11:44:20.029398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ecb0abea-5208-453b-86e2-58b64b0634ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":20984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":20984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":20984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":20984}]},"public_addr":"172.21.15.112:6804/20984","cluster_addr":"172.21.15.112:6805/20984","heartbeat_back_addr":"172.21.15.112:6807/20984","heartbeat_front_addr":"172.21.15.112:6806/20984","state":["exists","up"]},{"osd":1,"uuid":"3164789a-58af-4cf7-bd86-260d70dd02c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":20985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":20985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":20985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":20985}]},"public_addr":"172.21.15.112:6800/20985","cluster_addr":"172.21.15.112:6801/20985","heartbeat_back_addr":"172.21.15.112:6803/20985","heartbeat_front_addr":"172.21.15.112:6802/20985","state":["exists","up"]},{"osd":2,"uuid":"0223fabc-71f3-4642-8818-d99dbd5ba501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":20986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":20986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":20986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":20986}]},"public_addr":"172.21.15.112:6808/20986","cluster_addr":"172.21.15.112:6809/20986","heartbeat_back_addr":"172.21.15.112:6811/20986","heartbeat_front_addr":"172.21.15.112:6810/20986","state":["exists","up"]},{"osd":3,"uuid":"07461389-250c-4e30-8890-4267bddeac4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6804","nonce":21049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6805","nonce":21049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6807","nonce":21049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6806","nonce":21049}]},"public_addr":"172.21.15.129:6804/21049","cluster_addr":"172.21.15.129:6805/21049","heartbeat_back_addr":"172.21.15.129:6807/21049","heartbeat_front_addr":"172.21.15.129:6806/21049","state":["exists","up"]},{"osd":4,"uuid":"ad3a24d7-0278-4137-a499-e7b7337e1698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6808","nonce":21047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6809","nonce":21047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6811","nonce":21047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6810","nonce":21047}]},"public_addr":"172.21.15.129:6808/21047","cluster_addr":"172.21.15.129:6809/21047","heartbeat_back_addr":"172.21.15.129:6811/21047","heartbeat_front_addr":"172.21.15.129:6810/21047","state":["exists","up"]},{"osd":5,"uuid":"174b7f7f-23bc-45c9-9245-7c3edc4b6235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6800","nonce":21048}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6801","nonce":21048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6803","nonce":21048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6802","nonce":21048}]},"public_addr":"172.21.15.129:6800/21048","cluster_addr":"172.21.15.129:6801/21048","heartbeat_back_addr":"172.21.15.129:6803/21048","heartbeat_front_addr":"172.21.15.129:6802/21048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.965457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.813648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.196234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.055175+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.022483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.092059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:44:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 10 librados: watch_flush enter 2024-09-18T11:44:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 10 librados: watch_flush exit 2024-09-18T11:44:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 20 client.4245.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 10 client.4245.objecter successfully canceled tick 2024-09-18T11:44:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 >> v1:172.21.15.129:6812/20806 conn(0x7f21b00383e0 legacy=0x7f21b003a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 10 monclient: shutdown 2024-09-18T11:44:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 >> v1:172.21.15.112:6789/0 conn(0x7f21c413ca20 legacy=0x7f21c414e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 shutdown_connections 2024-09-18T11:44:25.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 >> 172.21.15.112:0/347654350 conn(0x7f21c4067da0 msgr2=0x7f21c409a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 shutdown_connections 2024-09-18T11:44:25.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 1 -- 172.21.15.112:0/347654350 wait complete. 2024-09-18T11:44:25.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.190+0000 7f21c880e700 1 librados: shutdown 2024-09-18T11:44:25.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:44:25.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T11:44:25.211 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:44:25.211 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T11:44:25.211 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-18T11:44:25.212 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-18T11:44:25.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient: monmap: 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.308964+0000 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.308964+0000 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.309 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 1 Processor -- start 2024-09-18T11:44:25.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 1 -- start start 2024-09-18T11:44:25.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient: init 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient(hunting): picked mon.noname-b con 0x7f330c0a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient(hunting): picked mon.noname-a con 0x7f330c0a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient(hunting): picked mon.noname-c con 0x7f330c1385b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c097270 con 0x7f330c0a0750 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c0a3170 con 0x7f330c1385b0 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c096cb0 con 0x7f330c0a0380 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3311ecf700 10 monclient(hunting): authenticate will time out at 837.150254s 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3310ecd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f330c0a0380 0x7f330c097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41336/0 (socket says 172.21.15.112:41336) 2024-09-18T11:44:25.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.306+0000 7f3310ecd700 1 -- 172.21.15.112:0/1694471381 learned_addr learned my addr 172.21.15.112:0/1694471381 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3310ecd700 1 -- 172.21.15.112:0/1694471381 >> v1:172.21.15.112:6790/0 conn(0x7f330c0a0380 legacy=0x7f330c097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330bfff700 1 -- 172.21.15.112:0/1694471381 >> v1:172.21.15.112:6789/0 conn(0x7f330c0a0750 legacy=0x7f330c0a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989703393 0 0) 0x7f330c096cb0 con 0x7f330c0a0380 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f33116ce700 1 -- 172.21.15.112:0/1694471381 >> v1:172.21.15.129:6789/0 conn(0x7f330c1385b0 legacy=0x7f330c13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): my global_id is 4250 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32fc003570 con 0x7f330c0a0380 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278670534 0 0) 0x7f330c097270 con 0x7f330c0a0750 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): my global_id is 4248 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f330c096cb0 con 0x7f330c0a0750 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132811260 0 0) 0x7f330c0a3170 con 0x7f330c1385b0 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient(hunting): my global_id is 4243 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f330c097270 con 0x7f330c1385b0 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088785960 0 0) 0x7f32fc003570 con 0x7f330c0a0380 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 >> v1:172.21.15.129:6789/0 conn(0x7f330c1385b0 legacy=0x7f330c13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 >> v1:172.21.15.112:6789/0 conn(0x7f330c0a0750 legacy=0x7f330c0a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 monclient: found mon.noname-b 2024-09-18T11:44:25.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0a1420 con 0x7f330c0a0380 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3300001940 con 0x7f330c0a0380 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 5 monclient: authenticate success, global_id 4250 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: dump: 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.316 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3300001bd0 con 0x7f330c0a0380 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 1 -- 172.21.15.112:0/1694471381 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3300004a90 con 0x7f330c0a0380 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f330b7fe700 10 monclient: dump: 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 4 set_mon_vals no callback set 2024-09-18T11:44:25.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 10 monclient: shutdown 2024-09-18T11:44:25.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 1 -- 172.21.15.112:0/1694471381 >> v1:172.21.15.112:6790/0 conn(0x7f330c0a0380 legacy=0x7f330c097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 1 -- 172.21.15.112:0/1694471381 shutdown_connections 2024-09-18T11:44:25.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 1 -- 172.21.15.112:0/1694471381 >> 172.21.15.112:0/1694471381 conn(0x7f330c0a5990 msgr2=0x7f330c0a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 1 -- 172.21.15.112:0/1694471381 shutdown_connections 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.310+0000 7f3311ecf700 1 -- 172.21.15.112:0/1694471381 wait complete. 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient: monmap: 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 librados: starting msgr at 2024-09-18T11:44:25.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 librados: starting objecter 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 Processor -- start 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 -- start start 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 librados: setting wanted keys 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 librados: calling monclient init 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient: init 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient(hunting): picked mon.noname-a con 0x7f330c0a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient(hunting): picked mon.noname-b con 0x7f330c0a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient(hunting): picked mon.noname-c con 0x7f330c1385b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c09efe0 con 0x7f330c0a0380 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c0983d0 con 0x7f330c0a0750 2024-09-18T11:44:25.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c0a2010 con 0x7f330c1385b0 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3311ecf700 10 monclient(hunting): authenticate will time out at 837.157532s 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f33116ce700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f330c1385b0 0x7f330c1497e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41344/0 (socket says 172.21.15.112:41344) 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f33116ce700 1 -- 172.21.15.112:0/504045955 learned_addr learned my addr 172.21.15.112:0/504045955 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f3310ecd700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6789/0 conn(0x7f330c0a0380 legacy=0x7f330c141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.314+0000 7f33116ce700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6790/0 conn(0x7f330c1385b0 legacy=0x7f330c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.318+0000 7f3310ecd700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6789/0 conn(0x7f330c0a0380 legacy=0x7f330c141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.318+0000 7f3310ecd700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6789/0 conn(0x7f330c0a0380 legacy=0x7f330c141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.318+0000 7f3310ecd700 1 --1- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6789/0 conn(0x7f330c0a0380 0x7f330c141470 :-1 s=OPENED pgs=74 cs=1 l=1).handle_message read tag failed 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.318+0000 7f3310ecd700 1 --1- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6789/0 conn(0x7f330c0a0380 0x7f330c141470 :-1 s=OPENED pgs=74 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:44:25.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.318+0000 7f330bfff700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.129:6789/0 conn(0x7f330c0a0750 legacy=0x7f330c09de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 client.?.objecter ms_handle_connect 0x7f330c0a0380 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 client.?.objecter ms_handle_connect 0x7f330c1385b0 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6789/0 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 client.?.objecter ms_handle_connect 0x7f330c0a0750 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267139181 0 0) 0x7f330c0a2010 con 0x7f330c1385b0 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): my global_id is 4253 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f0004f10 con 0x7f330c1385b0 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802959394 0 0) 0x7f330c0983d0 con 0x7f330c0a0750 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): my global_id is 4246 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f330c0a2010 con 0x7f330c0a0750 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 641748891 0 0) 0x7f32f0004f10 con 0x7f330c1385b0 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f330c0983d0 con 0x7f330c1385b0 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3304002010 con 0x7f330c1385b0 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): dump: 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.326 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.327 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.327 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.327 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.327 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026149720 0 0) 0x7f330c0a2010 con 0x7f330c0a0750 2024-09-18T11:44:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32f0004f10 con 0x7f330c0a0750 2024-09-18T11:44:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f32f8001bd0 con 0x7f330c0a0750 2024-09-18T11:44:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient(hunting): dump: 2024-09-18T11:44:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1408583868 0 0) 0x7f330c0983d0 con 0x7f330c1385b0 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.129:6789/0 conn(0x7f330c0a0750 legacy=0x7f330c09de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6789/0 conn(0x7f330c0a0380 legacy=0x7f330c141470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 monclient: found mon.c 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c137190 con 0x7f330c1385b0 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 5 monclient: authenticate success, global_id 4253 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 10 monclient: _renew_subs 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c098060 con 0x7f330c1385b0 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 10 monclient: _renew_subs 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c141b80 con 0x7f330c1385b0 2024-09-18T11:44:25.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 1 librados: init done 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3304002630 con 0x7f330c1385b0 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f330affd700 4 set_mon_vals no callback set 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3304002980 con 0x7f330c1385b0 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 10 monclient: dump: 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.332 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f3304002be0 con 0x7f330c1385b0 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 10 client.4253.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 15 client.4253.objecter _session_command_op_assign -1 1 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 20 client.4253.objecter _get_session osd=-1 returning homeless 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.112:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32ec0028c0 con 0x7f330c1385b0 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f3304043b50 con 0x7f330c1385b0 2024-09-18T11:44:25.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 10 client.4253.objecter ms_dispatch 0x7f330c000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 3 client.4253.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 3 client.4253.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 10 client.4253.objecter checking command 1 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 20 client.4253.objecter _get_session s=0x7f3304002630 osd=2 3 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 20 client.4253.objecter put_session s=0x7f3304002630 osd=2 3 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 15 client.4253.objecter _session_command_op_remove -1 1 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 20 client.4253.objecter _get_session s=0x7f3304002630 osd=2 3 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 20 client.4253.objecter get_session s=0x7f3304002630 osd=2 3 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 15 client.4253.objecter _session_command_op_assign 2 1 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 20 client.4253.objecter put_session s=0x7f3304002630 osd=2 4 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 10 client.4253.objecter _send_command 1 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.112:6808/20986 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32f0042b80 con 0x7f32f003eee0 2024-09-18T11:44:25.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 20 client.4253.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (unknown 391748328 0 0) 0x7f3304043e80 con 0x7f330c1385b0 2024-09-18T11:44:25.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T11:44:25.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3310ecd700 10 monclient: get_auth_request con 0x7f32f0038410 auth_method 0 2024-09-18T11:44:25.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f330bfff700 10 monclient: get_auth_request con 0x7f32f003eee0 auth_method 0 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7fabe1e97700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7fabe1e97700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.322+0000 7fabe1e97700 10 monclient: monmap: 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.328232+0000 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.328232+0000 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.336 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 1 Processor -- start 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 1 -- start start 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 10 monclient: init 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 10 monclient(hunting): picked mon.noname-b con 0x7fabdc132170 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 10 monclient(hunting): picked mon.noname-c con 0x7fabdc12edc0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 10 monclient(hunting): picked mon.noname-a con 0x7fabdc12f900 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc06d060 con 0x7fabdc12f900 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc132d20 con 0x7fabdc12edc0 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc06e680 con 0x7fabdc132170 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1e97700 10 monclient(hunting): authenticate will time out at 837.169855s 2024-09-18T11:44:25.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe0e95700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fabdc132170 0x7fabdc132610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41352/0 (socket says 172.21.15.112:41352) 2024-09-18T11:44:25.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe0e95700 1 -- 172.21.15.112:0/3362302884 learned_addr learned my addr 172.21.15.112:0/3362302884 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7fabe1696700 1 --1- 172.21.15.112:0/3362302884 >> v1:172.21.15.112:6789/0 conn(0x7fabdc12f900 0x7fabdc13a5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35054/0 (socket says 172.21.15.112:35054) 2024-09-18T11:44:25.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.326+0000 7f3308ff9700 10 client.4253.objecter ms_handle_connect 0x7f32f0038410 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabe0e95700 1 -- 172.21.15.112:0/3362302884 >> v1:172.21.15.112:6790/0 conn(0x7fabdc132170 legacy=0x7fabdc132610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabe1696700 1 -- 172.21.15.112:0/3362302884 >> v1:172.21.15.112:6789/0 conn(0x7fabdc12f900 legacy=0x7fabdc13a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367556750 0 0) 0x7fabdc06e680 con 0x7fabdc132170 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): my global_id is 4256 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdbfff700 1 -- 172.21.15.112:0/3362302884 >> v1:172.21.15.129:6789/0 conn(0x7fabdc12edc0 legacy=0x7fabdc12f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc4003570 con 0x7fabdc132170 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269295280 0 0) 0x7fabdc06d060 con 0x7fabdc12f900 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): my global_id is 4251 2024-09-18T11:44:25.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabdc06e680 con 0x7fabdc12f900 2024-09-18T11:44:25.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015093294 0 0) 0x7fabdc132d20 con 0x7fabdc12edc0 2024-09-18T11:44:25.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient(hunting): my global_id is 4249 2024-09-18T11:44:25.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabdc06d060 con 0x7fabdc12edc0 2024-09-18T11:44:25.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7f3308ff9700 10 client.4253.objecter ms_handle_connect 0x7f32f003eee0 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336723459 0 0) 0x7fabdc06e680 con 0x7fabdc12f900 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 >> v1:172.21.15.112:6790/0 conn(0x7fabdc132170 legacy=0x7fabdc132610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 >> v1:172.21.15.129:6789/0 conn(0x7fabdc12edc0 legacy=0x7fabdc12f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 monclient: found mon.noname-a 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc06e960 con 0x7fabdc12f900 2024-09-18T11:44:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fabcc001920 con 0x7fabdc12f900 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: dump: 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.342 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabcc001bb0 con 0x7fabdc12f900 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 1 -- 172.21.15.112:0/3362302884 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fabcc001da0 con 0x7fabdc12f900 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabdb7fe700 10 monclient: dump: 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.343 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabe1e97700 5 monclient: authenticate success, global_id 4251 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabe1e97700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabe1e97700 4 set_mon_vals no callback set 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== osd.2 v1:172.21.15.112:6808/20986 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f32f0042b80 con 0x7f32f003eee0 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7f3308ff9700 10 client.4253.objecter ms_dispatch 0x7f330c000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T11:44:25.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7f3308ff9700 10 client.4253.objecter _finish_command 1 = system:0 2024-09-18T11:44:25.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7f3308ff9700 20 client.4253.objecter put_session s=0x7f3304002630 osd=2 4 2024-09-18T11:44:25.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7f3308ff9700 15 client.4253.objecter _session_command_op_remove 2 1 2024-09-18T11:44:25.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabe1e97700 10 monclient: shutdown 2024-09-18T11:44:25.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabe1e97700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.330+0000 7fabe1e97700 1 -- 172.21.15.112:0/3362302884 >> v1:172.21.15.112:6789/0 conn(0x7fabdc12f900 legacy=0x7fabdc13a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 -- 172.21.15.112:0/3362302884 shutdown_connections 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 -- 172.21.15.112:0/3362302884 >> 172.21.15.112:0/3362302884 conn(0x7fabdc067da0 msgr2=0x7fabdc0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 -- 172.21.15.112:0/3362302884 shutdown_connections 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 -- 172.21.15.112:0/3362302884 wait complete. 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 10 monclient: monmap: 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.347 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 librados: starting msgr at 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 librados: starting objecter 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 Processor -- start 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 -- start start 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 librados: setting wanted keys 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 1 librados: calling monclient init 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.334+0000 7fabe1e97700 10 monclient: init 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 10 monclient(hunting): picked mon.noname-c con 0x7fabdc12edc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 10 monclient(hunting): picked mon.noname-b con 0x7fabdc12f900 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 10 monclient(hunting): picked mon.noname-a con 0x7fabdc132170 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc06d9d0 con 0x7fabdc132170 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc0a19f0 con 0x7fabdc12f900 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc1424c0 con 0x7fabdc12edc0 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1e97700 10 monclient(hunting): authenticate will time out at 837.180211s 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1696700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fabdc132170 0x7fabdc1499b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35056/0 (socket says 172.21.15.112:35056) 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1696700 1 -- 172.21.15.112:0/1547434451 learned_addr learned my addr 172.21.15.112:0/1547434451 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe1696700 1 -- 172.21.15.112:0/1547434451 >> v1:172.21.15.112:6789/0 conn(0x7fabdc132170 legacy=0x7fabdc1499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabd8ff9700 10 client.?.objecter ms_handle_connect 0x7fabdc132170 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabd8ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabd8ff9700 10 client.?.objecter ms_handle_connect 0x7fabdc12edc0 2024-09-18T11:44:25.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabd8ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabe0e95700 1 -- 172.21.15.112:0/1547434451 >> v1:172.21.15.112:6790/0 conn(0x7fabdc12edc0 legacy=0x7fabdc141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabdbfff700 1 -- 172.21.15.112:0/1547434451 >> v1:172.21.15.129:6789/0 conn(0x7fabdc12f900 legacy=0x7fabdc13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 DispatchQueue::entry inject delay of 0.002000 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7f3585c05700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7f3585c05700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.338+0000 7f3585c05700 10 monclient: monmap: 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.350 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.344163+0000 2024-09-18T11:44:25.351 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.344163+0000 2024-09-18T11:44:25.351 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.351 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.351 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.351 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.351 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.351 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 client.?.objecter ms_handle_connect 0x7fabdc12f900 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645108056 0 0) 0x7fabdc06d9d0 con 0x7fabdc132170 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): my global_id is 4254 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc8004f10 con 0x7fabdc132170 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873353665 0 0) 0x7fabdc1424c0 con 0x7fabdc12edc0 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): my global_id is 4259 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabdc06d9d0 con 0x7fabdc12edc0 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425320073 0 0) 0x7fabdc0a19f0 con 0x7fabdc12f900 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): my global_id is 4252 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabdc1424c0 con 0x7fabdc12f900 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 348344217 0 0) 0x7fabdc06d9d0 con 0x7fabdc12edc0 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabdc0a19f0 con 0x7fabdc12edc0 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644967123 0 0) 0x7fabc8004f10 con 0x7fabdc132170 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabdc06d9d0 con 0x7fabdc132170 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fabd0004830 con 0x7fabdc12edc0 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): dump: 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fabcc004b30 con 0x7fabdc132170 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient(hunting): dump: 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 160262383 0 0) 0x7fabdc0a19f0 con 0x7fabdc12edc0 2024-09-18T11:44:25.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 >> v1:172.21.15.129:6789/0 conn(0x7fabdc12f900 legacy=0x7fabdc13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 >> v1:172.21.15.112:6789/0 conn(0x7fabdc132170 legacy=0x7fabdc1499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 monclient: found mon.c 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc12d1c0 con 0x7fabdc12edc0 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 5 monclient: authenticate success, global_id 4259 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 10 monclient: _renew_subs 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc12e770 con 0x7fabdc12edc0 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 10 monclient: _renew_subs 2024-09-18T11:44:25.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc12d880 con 0x7fabdc12edc0 2024-09-18T11:44:25.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 1 librados: init done 2024-09-18T11:44:25.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 1 Processor -- start 2024-09-18T11:44:25.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 1 -- start start 2024-09-18T11:44:25.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 10 monclient: init 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd0001930 con 0x7fabdc12edc0 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fabd0004de0 con 0x7fabdc12edc0 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabdaffd700 4 set_mon_vals no callback set 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 monclient: dump: 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.359 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fabd000f660 con 0x7fabdc12edc0 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f09293b9700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f09293b9700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f09293b9700 10 monclient: monmap: 2024-09-18T11:44:25.360 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.346226+0000 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.346226+0000 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 10 monclient(hunting): picked mon.noname-b con 0x7f358006e0a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 10 monclient(hunting): picked mon.noname-a con 0x7f35800a1190 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 10 monclient(hunting): picked mon.noname-c con 0x7f35800a06f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35800a0b10 con 0x7f35800a1190 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358006b7c0 con 0x7f35800a06f0 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358006ce80 con 0x7f358006e0a0 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f3585c05700 10 monclient(hunting): authenticate will time out at 837.186795s 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7fabd0043670 con 0x7fabdc12edc0 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 10 client.4259.objecter ms_dispatch 0x7fabdc000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 3 client.4259.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 3 client.4259.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabd8ff9700 20 client.4259.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 10 client.4259.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 15 client.4259.objecter _session_command_op_assign -1 1 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 20 client.4259.objecter _get_session s=0x7fabbc002880 osd=5 3 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 20 client.4259.objecter put_session s=0x7fabbc002880 osd=5 3 2024-09-18T11:44:25.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 20 client.4259.objecter _get_session s=0x7fabbc002880 osd=5 3 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 15 client.4259.objecter _session_command_op_remove -1 1 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 20 client.4259.objecter get_session s=0x7fabbc002880 osd=5 3 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 15 client.4259.objecter _session_command_op_assign 5 1 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 20 client.4259.objecter put_session s=0x7fabbc002880 osd=5 4 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 10 client.4259.objecter _send_command 1 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.129:6800/21048 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabbc006cd0 con 0x7fabbc002f70 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabdbfff700 10 monclient: get_auth_request con 0x7fabc80384e0 auth_method 0 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f357ffff700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f35800a06f0 0x7f358013e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.112:36588/0 (socket says 172.21.15.112:36588) 2024-09-18T11:44:25.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f357ffff700 1 -- 172.21.15.112:0/2678791293 learned_addr learned my addr 172.21.15.112:0/2678791293 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fabe1696700 10 monclient: get_auth_request con 0x7fabbc002f70 auth_method 0 2024-09-18T11:44:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f995f221700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f995f221700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7f995f221700 10 monclient: monmap: 2024-09-18T11:44:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.365 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.344520+0000 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.344520+0000 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 1 Processor -- start 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 1 -- start start 2024-09-18T11:44:25.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 10 monclient: init 2024-09-18T11:44:25.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient: monmap: 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.344520+0000 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.344520+0000 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 1 Processor -- start 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 1 -- start start 2024-09-18T11:44:25.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient: init 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient(hunting): picked mon.noname-b con 0x7fa6980a21c0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient(hunting): picked mon.noname-c con 0x7fa69806ea70 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient(hunting): picked mon.noname-a con 0x7fa69809bcc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69806b860 con 0x7fa69809bcc0 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6980a2590 con 0x7fa69806ea70 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69806b290 con 0x7fa6980a21c0 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69f683700 10 monclient(hunting): authenticate will time out at 837.186953s 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69d41f700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa6980a21c0 0x7fa69806e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41376/0 (socket says 172.21.15.112:41376) 2024-09-18T11:44:25.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.342+0000 7fa69d41f700 1 -- 172.21.15.112:0/160974440 learned_addr learned my addr 172.21.15.112:0/160974440 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 10 monclient(hunting): picked mon.noname-b con 0x7f995809ae70 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 10 monclient(hunting): picked mon.noname-c con 0x7f9958132860 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 10 monclient(hunting): picked mon.noname-a con 0x7f9958132c60 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995806e750 con 0x7f9958132c60 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995809b240 con 0x7f9958132860 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995809b660 con 0x7f995809ae70 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995f221700 10 monclient(hunting): authenticate will time out at 837.188815s 2024-09-18T11:44:25.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357f7fe700 1 -- 172.21.15.112:0/2678791293 >> v1:172.21.15.112:6790/0 conn(0x7f358006e0a0 legacy=0x7f358006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357effd700 1 -- 172.21.15.112:0/2678791293 >> v1:172.21.15.112:6789/0 conn(0x7f35800a1190 legacy=0x7f358006c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69dc20700 1 -- 172.21.15.112:0/160974440 >> v1:172.21.15.112:6789/0 conn(0x7fa69809bcc0 legacy=0x7fa698147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995cfbd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f995809ae70 0x7f9958132150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41398/0 (socket says 172.21.15.112:41398) 2024-09-18T11:44:25.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995cfbd700 1 -- 172.21.15.112:0/831954934 learned_addr learned my addr 172.21.15.112:0/831954934 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69d41f700 1 -- 172.21.15.112:0/160974440 >> v1:172.21.15.112:6790/0 conn(0x7fa6980a21c0 legacy=0x7fa69806e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69dc20700 1 -- 172.21.15.112:0/160974440 >> v1:172.21.15.112:6789/0 conn(0x7fa69809bcc0 legacy=0x7fa698147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69dc20700 1 -- 172.21.15.112:0/160974440 >> v1:172.21.15.112:6789/0 conn(0x7fa69809bcc0 legacy=0x7fa698147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69dc20700 1 --1- 172.21.15.112:0/160974440 >> v1:172.21.15.112:6789/0 conn(0x7fa69809bcc0 0x7fa698147700 :-1 s=OPENED pgs=77 cs=1 l=1).handle_message read tag failed 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69dc20700 1 --1- 172.21.15.112:0/160974440 >> v1:172.21.15.112:6789/0 conn(0x7fa69809bcc0 0x7fa698147700 :-1 s=OPENED pgs=77 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6789/0 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69cc1e700 1 -- 172.21.15.112:0/160974440 >> v1:172.21.15.129:6789/0 conn(0x7fa69806ea70 legacy=0x7fa698143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249059444 0 0) 0x7fa69806b290 con 0x7fa6980a21c0 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient(hunting): my global_id is 4265 2024-09-18T11:44:25.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa684003570 con 0x7fa6980a21c0 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357ffff700 1 -- 172.21.15.112:0/2678791293 >> v1:172.21.15.129:6789/0 conn(0x7f35800a06f0 legacy=0x7f358013e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343784395 0 0) 0x7f358006ce80 con 0x7f358006e0a0 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient(hunting): my global_id is 4262 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3568003570 con 0x7f358006e0a0 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79345222 0 0) 0x7f35800a0b10 con 0x7f35800a1190 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient(hunting): my global_id is 4257 2024-09-18T11:44:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358006ce80 con 0x7f35800a1190 2024-09-18T11:44:25.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fabd8ff9700 10 client.4259.objecter ms_handle_connect 0x7fabc80384e0 2024-09-18T11:44:25.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798068332 0 0) 0x7fa6980a2590 con 0x7fa69806ea70 2024-09-18T11:44:25.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient(hunting): my global_id is 4255 2024-09-18T11:44:25.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69806b290 con 0x7fa69806ea70 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677092901 0 0) 0x7f3568003570 con 0x7f358006e0a0 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 >> v1:172.21.15.129:6789/0 conn(0x7f35800a06f0 legacy=0x7f358013e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 >> v1:172.21.15.112:6789/0 conn(0x7f35800a1190 legacy=0x7f358006c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 monclient: found mon.noname-b 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358006b4e0 con 0x7f358006e0a0 2024-09-18T11:44:25.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3570001990 con 0x7f358006e0a0 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: dump: 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.377 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.378 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3585c05700 5 monclient: authenticate success, global_id 4262 2024-09-18T11:44:25.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3585c05700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802779981 0 0) 0x7fa684003570 con 0x7fa6980a21c0 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 >> v1:172.21.15.129:6789/0 conn(0x7fa69806ea70 legacy=0x7fa698143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 >> v1:172.21.15.112:6789/0 conn(0x7fa69809bcc0 legacy=0x7fa698147700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 monclient: found mon.noname-b 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69806b580 con 0x7fa6980a21c0 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa694001130 con 0x7fa6980a21c0 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: dump: 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 5 monclient: authenticate success, global_id 4265 2024-09-18T11:44:25.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 1 Processor -- start 2024-09-18T11:44:25.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 10 client.4253.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-18T11:44:25.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 15 client.4253.objecter _session_command_op_assign -1 2 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 20 client.4253.objecter _get_session s=0x7f3304002630 osd=2 3 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 20 client.4253.objecter put_session s=0x7f3304002630 osd=2 3 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 20 client.4253.objecter _get_session s=0x7f3304002630 osd=2 3 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 15 client.4253.objecter _session_command_op_remove -1 2 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 20 client.4253.objecter get_session s=0x7f3304002630 osd=2 3 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 15 client.4253.objecter _session_command_op_assign 2 2 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 20 client.4253.objecter put_session s=0x7f3304002630 osd=2 4 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 10 client.4253.objecter _send_command 2 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 --> v1:172.21.15.112:6808/20986 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f32ec0011c0 con 0x7f32f003eee0 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 1 -- start start 2024-09-18T11:44:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 10 monclient: init 2024-09-18T11:44:25.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6940013c0 con 0x7fa6980a21c0 2024-09-18T11:44:25.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 1 -- 172.21.15.112:0/160974440 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa6940048f0 con 0x7fa6980a21c0 2024-09-18T11:44:25.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 4 set_mon_vals no callback set 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa68ffff700 10 monclient: dump: 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3308ff9700 1 -- 172.21.15.112:0/504045955 <== osd.2 v1:172.21.15.112:6808/20986 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3543570501) 0x7f32ec0011c0 con 0x7f32f003eee0 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3308ff9700 10 client.4253.objecter ms_dispatch 0x7f330c000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3308ff9700 10 client.4253.objecter _finish_command 2 = system:0 2024-09-18T11:44:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3308ff9700 20 client.4253.objecter put_session s=0x7f3304002630 osd=2 4 2024-09-18T11:44:25.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3308ff9700 15 client.4253.objecter _session_command_op_remove 2 2 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 10 monclient(hunting): picked mon.noname-b con 0x7f092412e180 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 10 monclient(hunting): picked mon.noname-a con 0x7f092406eef0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 10 monclient(hunting): picked mon.noname-c con 0x7f092412e7c0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092406f2e0 con 0x7f092406eef0 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092412ec20 con 0x7f092412e7c0 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092406ba70 con 0x7f092412e180 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f09293b9700 10 monclient(hunting): authenticate will time out at 837.191142s 2024-09-18T11:44:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f0922ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f092412e180 0x7f092406e7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41410/0 (socket says 172.21.15.112:41410) 2024-09-18T11:44:25.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3570001c20 con 0x7f358006e0a0 2024-09-18T11:44:25.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 1 -- 172.21.15.112:0/2678791293 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f35700020a0 con 0x7f358006e0a0 2024-09-18T11:44:25.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3585c05700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3585c05700 4 set_mon_vals no callback set 2024-09-18T11:44:25.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f0922ffd700 1 -- 172.21.15.112:0/3092244823 learned_addr learned my addr 172.21.15.112:0/3092244823 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f357e7fc700 10 monclient: dump: 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.389 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 10 monclient: shutdown 2024-09-18T11:44:25.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 1 -- 172.21.15.112:0/160974440 >> v1:172.21.15.112:6790/0 conn(0x7fa6980a21c0 legacy=0x7fa69806e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 10 librados: watch_flush enter 2024-09-18T11:44:25.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 10 librados: watch_flush exit 2024-09-18T11:44:25.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 10 client.4253.objecter close_session for osd.2 2024-09-18T11:44:25.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6808/20986 conn(0x7f32f003eee0 legacy=0x7f32f0041310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 1 -- 172.21.15.112:0/160974440 shutdown_connections 2024-09-18T11:44:25.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 1 -- 172.21.15.112:0/160974440 >> 172.21.15.112:0/160974440 conn(0x7fa6980680d0 msgr2=0x7fa6980a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 20 client.4253.objecter put_session s=0x7f3304002630 osd=2 1 2024-09-18T11:44:25.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 20 client.4253.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 10 client.4253.objecter successfully canceled tick 2024-09-18T11:44:25.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 1 -- 172.21.15.112:0/160974440 shutdown_connections 2024-09-18T11:44:25.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7fa69f683700 1 -- 172.21.15.112:0/160974440 wait complete. 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.129:6812/20806 conn(0x7f32f0038410 legacy=0x7f32f003a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 10 monclient: shutdown 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 >> v1:172.21.15.112:6790/0 conn(0x7f330c1385b0 legacy=0x7f330c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f33116ce700 1 -- 172.21.15.112:0/504045955 reap_dead start 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 shutdown_connections 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 >> 172.21.15.112:0/504045955 conn(0x7f330c0a5990 msgr2=0x7f330c138980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 shutdown_connections 2024-09-18T11:44:25.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f3311ecf700 1 -- 172.21.15.112:0/504045955 wait complete. 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995cfbd700 1 -- 172.21.15.112:0/831954934 >> v1:172.21.15.112:6790/0 conn(0x7f995809ae70 legacy=0x7f9958132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.346+0000 7f995d7be700 1 -- 172.21.15.112:0/831954934 >> v1:172.21.15.112:6789/0 conn(0x7f9958132c60 legacy=0x7f995813abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient: monmap: 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.394 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141850368 0 0) 0x7f995809b660 con 0x7f995809ae70 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient(hunting): my global_id is 4268 2024-09-18T11:44:25.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9948003570 con 0x7f995809ae70 2024-09-18T11:44:25.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f9957fff700 1 -- 172.21.15.112:0/831954934 >> v1:172.21.15.129:6789/0 conn(0x7f9958132860 legacy=0x7f9958137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716253983 0 0) 0x7f995806e750 con 0x7f9958132c60 2024-09-18T11:44:25.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient(hunting): my global_id is 4260 2024-09-18T11:44:25.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f995809b660 con 0x7f9958132c60 2024-09-18T11:44:25.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3311ecf700 1 librados: shutdown 2024-09-18T11:44:25.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 589722480 0 0) 0x7f9948003570 con 0x7f995809ae70 2024-09-18T11:44:25.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 >> v1:172.21.15.129:6789/0 conn(0x7f9958132860 legacy=0x7f9958137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 librados: starting msgr at 2024-09-18T11:44:25.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 librados: starting objecter 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 >> v1:172.21.15.112:6789/0 conn(0x7f9958132c60 legacy=0x7f995813abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 monclient: found mon.noname-b 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995809b8a0 con 0x7f995809ae70 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 5 monclient: authenticate success, global_id 4268 2024-09-18T11:44:25.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f994c001940 con 0x7f995809ae70 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: dump: 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.399 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 Processor -- start 2024-09-18T11:44:25.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fabd8ff9700 10 client.4259.objecter ms_handle_connect 0x7fabbc002f70 2024-09-18T11:44:25.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient: shutdown 2024-09-18T11:44:25.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- 172.21.15.112:0/2678791293 >> v1:172.21.15.112:6790/0 conn(0x7f358006e0a0 legacy=0x7f358006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- 172.21.15.112:0/2678791293 shutdown_connections 2024-09-18T11:44:25.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- 172.21.15.112:0/2678791293 >> 172.21.15.112:0/2678791293 conn(0x7f35800680d0 msgr2=0x7f35800a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 -- start start 2024-09-18T11:44:25.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 librados: setting wanted keys 2024-09-18T11:44:25.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 librados: calling monclient init 2024-09-18T11:44:25.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient: init 2024-09-18T11:44:25.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- 172.21.15.112:0/2678791293 shutdown_connections 2024-09-18T11:44:25.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f994c001bd0 con 0x7f995809ae70 2024-09-18T11:44:25.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 1 -- 172.21.15.112:0/831954934 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f994c004a90 con 0x7f995809ae70 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f99577fe700 10 monclient: dump: 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 4 set_mon_vals no callback set 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient(hunting): picked mon.noname-c con 0x7fa69806ea70 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient(hunting): picked mon.noname-a con 0x7fa69809bcc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient(hunting): picked mon.noname-b con 0x7fa6980a21c0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa698083120 con 0x7fa69809bcc0 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa698081ee0 con 0x7fa6980a21c0 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69809ace0 con 0x7fa69806ea70 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69f683700 10 monclient(hunting): authenticate will time out at 837.193208s 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69d41f700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa69806ea70 0x7fa698099ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41424/0 (socket says 172.21.15.112:41424) 2024-09-18T11:44:25.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69d41f700 1 -- 172.21.15.112:0/3963738824 learned_addr learned my addr 172.21.15.112:0/3963738824 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 10 monclient: shutdown 2024-09-18T11:44:25.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 1 -- 172.21.15.112:0/831954934 >> v1:172.21.15.112:6790/0 conn(0x7f995809ae70 legacy=0x7f9958132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- 172.21.15.112:0/2678791293 wait complete. 2024-09-18T11:44:25.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient: monmap: 2024-09-18T11:44:25.408 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.408 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.408 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.408 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.408 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 librados: starting msgr at 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 librados: starting objecter 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 1 -- 172.21.15.112:0/831954934 shutdown_connections 2024-09-18T11:44:25.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 1 -- 172.21.15.112:0/831954934 >> 172.21.15.112:0/831954934 conn(0x7f99580680d0 msgr2=0x7f99580a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 Processor -- start 2024-09-18T11:44:25.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0922ffd700 1 -- 172.21.15.112:0/3092244823 >> v1:172.21.15.112:6790/0 conn(0x7f092412e180 legacy=0x7f092406e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f09237fe700 1 -- 172.21.15.112:0/3092244823 >> v1:172.21.15.129:6789/0 conn(0x7f092412e7c0 legacy=0x7f0924136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- start start 2024-09-18T11:44:25.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 librados: setting wanted keys 2024-09-18T11:44:25.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 librados: calling monclient init 2024-09-18T11:44:25.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient: init 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908672553 0 0) 0x7f092412ec20 con 0x7f092412e7c0 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient(hunting): my global_id is 4264 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== osd.5 v1:172.21.15.129:6800/21048 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabbc006cd0 con 0x7fabbc002f70 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fabd8ff9700 10 client.4259.objecter ms_dispatch 0x7fabdc000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fabd8ff9700 10 client.4259.objecter _finish_command 1 = system:0 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fabd8ff9700 20 client.4259.objecter put_session s=0x7fabbc002880 osd=5 4 2024-09-18T11:44:25.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fabd8ff9700 15 client.4259.objecter _session_command_op_remove 5 1 2024-09-18T11:44:25.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c003570 con 0x7f092412e7c0 2024-09-18T11:44:25.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982292057 0 0) 0x7f092406ba70 con 0x7f092412e180 2024-09-18T11:44:25.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient(hunting): my global_id is 4271 2024-09-18T11:44:25.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 1 -- 172.21.15.112:0/831954934 shutdown_connections 2024-09-18T11:44:25.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f092412ec20 con 0x7f092412e180 2024-09-18T11:44:25.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 1 -- 172.21.15.112:0/831954934 wait complete. 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173849624 0 0) 0x7f092412ec20 con 0x7f092412e180 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 >> v1:172.21.15.129:6789/0 conn(0x7f092412e7c0 legacy=0x7f0924136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 >> v1:172.21.15.112:6789/0 conn(0x7f092406eef0 legacy=0x7f09241334b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 monclient: found mon.noname-b 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092412e550 con 0x7f092412e180 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0918001920 con 0x7f092412e180 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: dump: 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.415 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient(hunting): picked mon.noname-c con 0x7f35800a06f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient(hunting): picked mon.noname-b con 0x7f35800a1190 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient(hunting): picked mon.noname-a con 0x7f358011bb90 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358011da90 con 0x7f358011bb90 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358011d440 con 0x7f35800a1190 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358011e530 con 0x7f35800a06f0 2024-09-18T11:44:25.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f3585c05700 10 monclient(hunting): authenticate will time out at 837.194586s 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0918001bb0 con 0x7f092412e180 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 1 -- 172.21.15.112:0/3092244823 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0918004a90 con 0x7f092412e180 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f0921ffb700 10 monclient: dump: 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.417 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f357effd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f35800a1190 0x7f358014e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.112:36630/0 (socket says 172.21.15.112:36630) 2024-09-18T11:44:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f357effd700 1 -- 172.21.15.112:0/3187856659 learned_addr learned my addr 172.21.15.112:0/3187856659 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 10 monclient: monmap: 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 1 librados: starting msgr at 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 1 librados: starting objecter 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f995f221700 1 Processor -- start 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa69d41f700 1 -- 172.21.15.112:0/3963738824 >> v1:172.21.15.112:6790/0 conn(0x7fa69806ea70 legacy=0x7fa698099ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa68d7fa700 10 client.?.objecter ms_handle_connect 0x7fa69806ea70 2024-09-18T11:44:25.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa68d7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f09293b9700 5 monclient: authenticate success, global_id 4271 2024-09-18T11:44:25.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f09293b9700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7f09293b9700 4 set_mon_vals no callback set 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202587180 0 0) 0x7fa69809ace0 con 0x7fa69806ea70 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa68d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa68d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa68d7fa700 10 monclient(hunting): my global_id is 4274 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.350+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa67c004f10 con 0x7fa69806ea70 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 client.?.objecter ms_handle_connect 0x7fa6980a21c0 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69dc20700 1 -- 172.21.15.112:0/3963738824 >> v1:172.21.15.129:6789/0 conn(0x7fa6980a21c0 legacy=0x7fa698156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428514679 0 0) 0x7fa67c004f10 con 0x7fa69806ea70 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa69809ace0 con 0x7fa69806ea70 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 client.?.objecter ms_handle_connect 0x7fa69809bcc0 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa694004830 con 0x7fa69806ea70 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient(hunting): dump: 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.424 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69cc1e700 1 -- 172.21.15.112:0/3963738824 >> v1:172.21.15.112:6789/0 conn(0x7fa69809bcc0 legacy=0x7fa69809a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165305303 0 0) 0x7fa698081ee0 con 0x7fa6980a21c0 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient(hunting): my global_id is 4267 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa67c004f10 con 0x7fa6980a21c0 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 71288734 0 0) 0x7fa69809ace0 con 0x7fa69806ea70 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 >> v1:172.21.15.129:6789/0 conn(0x7fa6980a21c0 legacy=0x7fa698156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 >> v1:172.21.15.112:6789/0 conn(0x7fa69809bcc0 legacy=0x7fa69809a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 monclient: found mon.c 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698086390 con 0x7fa69806ea70 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 5 monclient: authenticate success, global_id 4274 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 10 monclient: _renew_subs 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa698083400 con 0x7fa69806ea70 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 10 monclient: _renew_subs 2024-09-18T11:44:25.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa698083d50 con 0x7fa69806ea70 2024-09-18T11:44:25.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69f683700 1 librados: init done 2024-09-18T11:44:25.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f09293b9700 10 monclient: shutdown 2024-09-18T11:44:25.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f09293b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f09293b9700 1 -- 172.21.15.112:0/3092244823 >> v1:172.21.15.112:6790/0 conn(0x7f092412e180 legacy=0x7f092406e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 1 -- start start 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 1 librados: setting wanted keys 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 1 librados: calling monclient init 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 10 monclient: init 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa694004f60 con 0x7fa69806ea70 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa694001da0 con 0x7fa69806ea70 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 monclient: dump: 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.428 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f357effd700 1 -- 172.21.15.112:0/3187856659 >> v1:172.21.15.129:6789/0 conn(0x7f35800a1190 legacy=0x7f358014e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa694005270 con 0x7fa69806ea70 2024-09-18T11:44:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f357f7fe700 1 -- 172.21.15.112:0/3187856659 >> v1:172.21.15.112:6790/0 conn(0x7f35800a06f0 legacy=0x7f358011de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7fa694043960 con 0x7fa69806ea70 2024-09-18T11:44:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 10 client.4274.objecter ms_dispatch 0x7fa698000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 3 client.4274.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 3 client.4274.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa68d7fa700 20 client.4274.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f357ffff700 1 -- 172.21.15.112:0/3187856659 >> v1:172.21.15.112:6789/0 conn(0x7f358011bb90 legacy=0x7f358014eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 10 monclient(hunting): picked mon.noname-b con 0x7f9958132860 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 10 monclient(hunting): picked mon.noname-c con 0x7f9958132c60 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 10 monclient(hunting): picked mon.noname-a con 0x7f9958147aa0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995811cf90 con 0x7f9958147aa0 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9958132310 con 0x7f9958132860 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99580a0ef0 con 0x7f9958132c60 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f995f221700 10 monclient(hunting): authenticate will time out at 837.197736s 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f9957fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f9958132c60 0x7f99581313b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41450/0 (socket says 172.21.15.112:41450) 2024-09-18T11:44:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f9957fff700 1 -- 172.21.15.112:0/815747978 learned_addr learned my addr 172.21.15.112:0/815747978 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7fa69cc1e700 10 monclient: get_auth_request con 0x7fa67c038210 auth_method 0 2024-09-18T11:44:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.354+0000 7f9957fff700 1 -- 172.21.15.112:0/815747978 >> v1:172.21.15.112:6790/0 conn(0x7f9958132c60 legacy=0x7f99581313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 client.?.objecter ms_handle_connect 0x7f9958132c60 2024-09-18T11:44:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa68effd700 4 set_mon_vals no callback set 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 15 client.4274.objecter _session_command_op_assign -1 1 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 20 client.4274.objecter _get_session s=0x7fa678002880 osd=1 3 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 20 client.4274.objecter put_session s=0x7fa678002880 osd=1 3 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 20 client.4274.objecter _get_session s=0x7fa678002880 osd=1 3 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 15 client.4274.objecter _session_command_op_remove -1 1 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 20 client.4274.objecter get_session s=0x7fa678002880 osd=1 3 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 15 client.4274.objecter _session_command_op_assign 1 1 2024-09-18T11:44:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 20 client.4274.objecter put_session s=0x7fa678002880 osd=1 4 2024-09-18T11:44:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 10 client.4274.objecter _send_command 1 2024-09-18T11:44:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 --> v1:172.21.15.112:6800/20985 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa678006cd0 con 0x7fa678002f70 2024-09-18T11:44:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428560241 0 0) 0x7f99580a0ef0 con 0x7f9958132c60 2024-09-18T11:44:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): my global_id is 4280 2024-09-18T11:44:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c004f10 con 0x7f9958132c60 2024-09-18T11:44:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995cfbd700 1 -- 172.21.15.112:0/815747978 >> v1:172.21.15.129:6789/0 conn(0x7f9958132860 legacy=0x7f9958141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 client.?.objecter ms_handle_connect 0x7f9958132860 2024-09-18T11:44:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa69dc20700 10 monclient: get_auth_request con 0x7fa678002f70 auth_method 0 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995d7be700 1 -- 172.21.15.112:0/815747978 >> v1:172.21.15.112:6789/0 conn(0x7f9958147aa0 legacy=0x7f9958149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 client.?.objecter ms_handle_connect 0x7f9958147aa0 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225102869 0 0) 0x7f9958132310 con 0x7f9958132860 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): my global_id is 4273 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99580a0ef0 con 0x7f9958132860 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1568890239 0 0) 0x7f993c004f10 con 0x7f9958132c60 2024-09-18T11:44:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9958132310 con 0x7f9958132c60 2024-09-18T11:44:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397949249 0 0) 0x7f995811cf90 con 0x7f9958147aa0 2024-09-18T11:44:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): my global_id is 4269 2024-09-18T11:44:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c004f10 con 0x7f9958147aa0 2024-09-18T11:44:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7fa68d7fa700 10 client.4274.objecter ms_handle_connect 0x7fa67c038210 2024-09-18T11:44:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9944001c80 con 0x7f9958132c60 2024-09-18T11:44:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): dump: 2024-09-18T11:44:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3311669226 0 0) 0x7f9958132310 con 0x7f9958132c60 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 >> v1:172.21.15.129:6789/0 conn(0x7f9958132860 legacy=0x7f9958141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 >> v1:172.21.15.112:6789/0 conn(0x7f9958147aa0 legacy=0x7f9958149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 monclient: found mon.c 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995811e4d0 con 0x7f9958132c60 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 5 monclient: authenticate success, global_id 4280 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 10 monclient: _renew_subs 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995811d5e0 con 0x7f9958132c60 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 10 monclient: _renew_subs 2024-09-18T11:44:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f995811ca10 con 0x7f9958132c60 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995f221700 1 librados: init done 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9944002230 con 0x7f9958132c60 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9944002640 con 0x7f9958132c60 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9956ffd700 4 set_mon_vals no callback set 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 monclient: dump: 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f9944002820 con 0x7f9958132c60 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f9944043800 con 0x7f9958132c60 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 10 client.4280.objecter ms_dispatch 0x7f9958000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 3 client.4280.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 3 client.4280.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f9954ff9700 20 client.4280.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.358+0000 7f995cfbd700 10 monclient: get_auth_request con 0x7f993c038480 auth_method 0 2024-09-18T11:44:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 client.?.objecter ms_handle_connect 0x7f35800a1190 2024-09-18T11:44:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 client.?.objecter ms_handle_connect 0x7f35800a06f0 2024-09-18T11:44:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 client.?.objecter ms_handle_connect 0x7f358011bb90 2024-09-18T11:44:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776687664 0 0) 0x7f358011d440 con 0x7f35800a1190 2024-09-18T11:44:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 -- 172.21.15.112:0/3092244823 shutdown_connections 2024-09-18T11:44:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 -- 172.21.15.112:0/3092244823 >> 172.21.15.112:0/3092244823 conn(0x7f09240680d0 msgr2=0x7f09240a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): my global_id is 4270 2024-09-18T11:44:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356c004f10 con 0x7f35800a1190 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194465890 0 0) 0x7f358011da90 con 0x7f358011bb90 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): my global_id is 4266 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358011d440 con 0x7f358011bb90 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650885056 0 0) 0x7f358011e530 con 0x7f35800a06f0 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): my global_id is 4277 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358011da90 con 0x7f35800a06f0 2024-09-18T11:44:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 -- 172.21.15.112:0/3092244823 shutdown_connections 2024-09-18T11:44:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 -- 172.21.15.112:0/3092244823 wait complete. 2024-09-18T11:44:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2263810466 0 0) 0x7f358011d440 con 0x7f358011bb90 2024-09-18T11:44:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f358011e530 con 0x7f358011bb90 2024-09-18T11:44:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 306349620 0 0) 0x7f358011da90 con 0x7f35800a06f0 2024-09-18T11:44:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7fa68d7fa700 10 client.4274.objecter ms_handle_connect 0x7fa678002f70 2024-09-18T11:44:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 15 client.4280.objecter _session_command_op_assign -1 1 2024-09-18T11:44:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 20 client.4280.objecter _get_session s=0x7f9938002880 osd=0 3 2024-09-18T11:44:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 20 client.4280.objecter put_session s=0x7f9938002880 osd=0 3 2024-09-18T11:44:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 20 client.4280.objecter _get_session s=0x7f9938002880 osd=0 3 2024-09-18T11:44:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 15 client.4280.objecter _session_command_op_remove -1 1 2024-09-18T11:44:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 20 client.4280.objecter get_session s=0x7f9938002880 osd=0 3 2024-09-18T11:44:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 15 client.4280.objecter _session_command_op_assign 0 1 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 20 client.4280.objecter put_session s=0x7f9938002880 osd=0 4 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 10 client.4280.objecter _send_command 1 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.112:6804/20984 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9938006cd0 con 0x7f9938002f70 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f995d7be700 10 monclient: get_auth_request con 0x7f9938002f70 auth_method 0 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient: monmap: 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.450 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.451 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.451 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.451 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.451 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.451 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.451 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.452 INFO:teuthology.orchestra.run.smithi112.stdout:51539607557 2024-09-18T11:44:25.452 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:44:25.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f9954ff9700 10 client.4280.objecter ms_handle_connect 0x7f993c038480 2024-09-18T11:44:25.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f358011d440 con 0x7f35800a06f0 2024-09-18T11:44:25.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 librados: starting msgr at 2024-09-18T11:44:25.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 librados: starting objecter 2024-09-18T11:44:25.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== osd.1 v1:172.21.15.112:6800/20985 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa678006cd0 con 0x7fa678002f70 2024-09-18T11:44:25.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7fa68d7fa700 10 client.4274.objecter ms_dispatch 0x7fa698000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T11:44:25.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7fa68d7fa700 10 client.4274.objecter _finish_command 1 = system:0 2024-09-18T11:44:25.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7fa68d7fa700 20 client.4274.objecter put_session s=0x7fa678002880 osd=1 4 2024-09-18T11:44:25.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7fa68d7fa700 15 client.4274.objecter _session_command_op_remove 1 1 2024-09-18T11:44:25.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4242416764 0 0) 0x7f356c004f10 con 0x7f35800a1190 2024-09-18T11:44:25.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f358011da90 con 0x7f35800a1190 2024-09-18T11:44:25.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3578001f30 con 0x7f358011bb90 2024-09-18T11:44:25.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): dump: 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f35700020a0 con 0x7f35800a06f0 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): dump: 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3574001ca0 con 0x7f35800a1190 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient(hunting): dump: 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.457 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 Processor -- start 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 -- start start 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 librados: setting wanted keys 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 librados: calling monclient init 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient: init 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient(hunting): picked mon.noname-b con 0x7f092406eef0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient(hunting): picked mon.noname-c con 0x7f092412e180 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient(hunting): picked mon.noname-a con 0x7f092412e7c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092411c170 con 0x7f092412e7c0 2024-09-18T11:44:25.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092409be00 con 0x7f092406eef0 2024-09-18T11:44:25.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092409edc0 con 0x7f092412e180 2024-09-18T11:44:25.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09293b9700 10 monclient(hunting): authenticate will time out at 837.206588s 2024-09-18T11:44:25.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09227fc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f092412e180 0x7f092413d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41460/0 (socket says 172.21.15.112:41460) 2024-09-18T11:44:25.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f09227fc700 1 -- 172.21.15.112:0/2743684919 learned_addr learned my addr 172.21.15.112:0/2743684919 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1424842587 0 0) 0x7f358011e530 con 0x7f358011bb90 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 >> v1:172.21.15.112:6790/0 conn(0x7f35800a06f0 legacy=0x7f358011de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 >> v1:172.21.15.129:6789/0 conn(0x7f35800a1190 legacy=0x7f358014e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 monclient: found mon.a 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358014f5f0 con 0x7f358011bb90 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 5 monclient: authenticate success, global_id 4266 2024-09-18T11:44:25.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 10 monclient: _renew_subs 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358011d160 con 0x7f358011bb90 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 10 monclient: _renew_subs 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3580157c50 con 0x7f358011bb90 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 1 librados: init done 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3578002090 con 0x7f358011bb90 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3578005000 con 0x7f358011bb90 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f357dffb700 4 set_mon_vals no callback set 2024-09-18T11:44:25.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 10 monclient: dump: 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 10 client.4266.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 15 client.4266.objecter _session_command_op_assign -1 1 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 20 client.4266.objecter _get_session osd=-1 returning homeless 2024-09-18T11:44:25.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.112:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f35600028c0 con 0x7f358011bb90 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.362+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f35780051d0 con 0x7f358011bb90 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f3578043810 con 0x7f358011bb90 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 10 client.4266.objecter ms_dispatch 0x7f3580000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 3 client.4266.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 3 client.4266.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 10 client.4266.objecter checking command 1 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 20 client.4266.objecter _get_session s=0x7f356c03eeb0 osd=3 3 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 20 client.4266.objecter put_session s=0x7f356c03eeb0 osd=3 3 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 15 client.4266.objecter _session_command_op_remove -1 1 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 20 client.4266.objecter _get_session s=0x7f356c03eeb0 osd=3 3 2024-09-18T11:44:25.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 20 client.4266.objecter get_session s=0x7f356c03eeb0 osd=3 3 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 15 client.4266.objecter _session_command_op_assign 3 1 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 20 client.4266.objecter put_session s=0x7f356c03eeb0 osd=3 4 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 10 client.4266.objecter _send_command 1 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.129:6804/21049 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f356c043200 con 0x7f356c03f560 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 20 client.4266.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (unknown 391748328 0 0) 0x7f3578043c40 con 0x7f358011bb90 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f357f7fe700 10 monclient: get_auth_request con 0x7f356c038920 auth_method 0 2024-09-18T11:44:25.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f357effd700 10 monclient: get_auth_request con 0x7f356c03f560 auth_method 0 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09227fc700 1 -- 172.21.15.112:0/2743684919 >> v1:172.21.15.112:6790/0 conn(0x7f092412e180 legacy=0x7f092413d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 client.?.objecter ms_handle_connect 0x7f092412e180 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035316420 0 0) 0x7f092409edc0 con 0x7f092412e180 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): my global_id is 4283 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c005660 con 0x7f092412e180 2024-09-18T11:44:25.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 client.?.objecter ms_handle_connect 0x7f092406eef0 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f0922ffd700 1 -- 172.21.15.112:0/2743684919 >> v1:172.21.15.129:6789/0 conn(0x7f092406eef0 legacy=0x7f092413d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 client.?.objecter ms_handle_connect 0x7f092412e7c0 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09237fe700 1 -- 172.21.15.112:0/2743684919 >> v1:172.21.15.112:6789/0 conn(0x7f092412e7c0 legacy=0x7f0924145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3806619001 0 0) 0x7f090c005660 con 0x7f092412e180 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f092409edc0 con 0x7f092412e180 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627750180 0 0) 0x7f092409be00 con 0x7f092406eef0 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): my global_id is 4276 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c005660 con 0x7f092406eef0 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0914002c40 con 0x7f092412e180 2024-09-18T11:44:25.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): dump: 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f9954ff9700 10 client.4280.objecter ms_handle_connect 0x7f9938002f70 2024-09-18T11:44:25.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561397461 0 0) 0x7f092411c170 con 0x7f092412e7c0 2024-09-18T11:44:25.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): my global_id is 4272 2024-09-18T11:44:25.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f092409be00 con 0x7f092412e7c0 2024-09-18T11:44:25.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1011731092 0 0) 0x7f090c005660 con 0x7f092406eef0 2024-09-18T11:44:25.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f092411c170 con 0x7f092406eef0 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0918004830 con 0x7f092406eef0 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): dump: 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.471 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3285291491 0 0) 0x7f092409be00 con 0x7f092412e7c0 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f090c005660 con 0x7f092412e7c0 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f091c001970 con 0x7f092412e7c0 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient(hunting): dump: 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.472 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2744143872 0 0) 0x7f090c005660 con 0x7f092412e7c0 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 >> v1:172.21.15.112:6790/0 conn(0x7f092412e180 legacy=0x7f092413d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 >> v1:172.21.15.129:6789/0 conn(0x7f092406eef0 legacy=0x7f092413d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 monclient: found mon.a 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092411bcc0 con 0x7f092412e7c0 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 5 monclient: authenticate success, global_id 4272 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 10 monclient: _renew_subs 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092413e160 con 0x7f092412e7c0 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 10 monclient: _renew_subs 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f092411b500 con 0x7f092412e7c0 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 1 librados: init done 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== osd.0 v1:172.21.15.112:6804/20984 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9938006cd0 con 0x7f9938002f70 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f9954ff9700 10 client.4280.objecter ms_dispatch 0x7f9958000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T11:44:25.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f9954ff9700 10 client.4280.objecter _finish_command 1 = system:0 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f9954ff9700 20 client.4280.objecter put_session s=0x7f9938002880 osd=0 4 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f9954ff9700 15 client.4280.objecter _session_command_op_remove 0 1 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f091c002530 con 0x7f092412e7c0 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f091c002840 con 0x7f092412e7c0 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 monclient: dump: 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.475 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.476 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.476 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.476 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.476 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.476 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.476 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f35677fe700 10 client.4266.objecter ms_handle_connect 0x7f356c038920 2024-09-18T11:44:25.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f091c002a50 con 0x7f092412e7c0 2024-09-18T11:44:25.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 10 client.4272.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 15 client.4272.objecter _session_command_op_assign -1 1 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 20 client.4272.objecter _get_session osd=-1 returning homeless 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.112:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f092405f2e0 con 0x7f092412e7c0 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f091c043980 con 0x7f092412e7c0 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 10 client.4272.objecter ms_dispatch 0x7f0924000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 3 client.4272.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.366+0000 7f090affd700 3 client.4272.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 10 client.4272.objecter checking command 1 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 20 client.4272.objecter _get_session s=0x7f090c03efe0 osd=4 3 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 20 client.4272.objecter put_session s=0x7f090c03efe0 osd=4 3 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 15 client.4272.objecter _session_command_op_remove -1 1 2024-09-18T11:44:25.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 20 client.4272.objecter _get_session s=0x7f090c03efe0 osd=4 3 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 20 client.4272.objecter get_session s=0x7f090c03efe0 osd=4 3 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 15 client.4272.objecter _session_command_op_assign 4 1 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 20 client.4272.objecter put_session s=0x7f090c03efe0 osd=4 4 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 10 client.4272.objecter _send_command 1 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.129:6808/21047 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f090c043330 con 0x7f090c03f690 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 20 client.4272.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (unknown 391748328 0 0) 0x7f091c0030e0 con 0x7f092412e7c0 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f0922ffd700 10 monclient: get_auth_request con 0x7f090c038b00 auth_method 0 2024-09-18T11:44:25.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f35677fe700 10 client.4266.objecter ms_handle_connect 0x7f356c03f560 2024-09-18T11:44:25.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f090affd700 10 client.4272.objecter ms_handle_connect 0x7f090c038b00 2024-09-18T11:44:25.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== osd.3 v1:172.21.15.129:6804/21049 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f356c043200 con 0x7f356c03f560 2024-09-18T11:44:25.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f35677fe700 10 client.4266.objecter ms_dispatch 0x7f3580000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T11:44:25.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f35677fe700 10 client.4266.objecter _finish_command 1 = system:0 2024-09-18T11:44:25.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f35677fe700 20 client.4266.objecter put_session s=0x7f356c03eeb0 osd=3 4 2024-09-18T11:44:25.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f35677fe700 15 client.4266.objecter _session_command_op_remove 3 1 2024-09-18T11:44:25.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7f09217fa700 4 set_mon_vals no callback set 2024-09-18T11:44:25.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 10 client.4259.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-18T11:44:25.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 15 client.4259.objecter _session_command_op_assign -1 2 2024-09-18T11:44:25.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 20 client.4259.objecter _get_session s=0x7fabbc002880 osd=5 3 2024-09-18T11:44:25.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 20 client.4259.objecter put_session s=0x7fabbc002880 osd=5 3 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 20 client.4259.objecter _get_session s=0x7fabbc002880 osd=5 3 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 15 client.4259.objecter _session_command_op_remove -1 2 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 20 client.4259.objecter get_session s=0x7fabbc002880 osd=5 3 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 15 client.4259.objecter _session_command_op_assign 5 2 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 20 client.4259.objecter put_session s=0x7fabbc002880 osd=5 4 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 10 client.4259.objecter _send_command 2 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 --> v1:172.21.15.129:6800/21048 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fabbc0011c0 con 0x7fabbc002f70 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabd8ff9700 1 -- 172.21.15.112:0/1547434451 <== osd.5 v1:172.21.15.129:6800/21048 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3543570501) 0x7fabbc0011c0 con 0x7fabbc002f70 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabd8ff9700 10 client.4259.objecter ms_dispatch 0x7fabdc000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabd8ff9700 10 client.4259.objecter _finish_command 2 = system:0 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabd8ff9700 20 client.4259.objecter put_session s=0x7fabbc002880 osd=5 4 2024-09-18T11:44:25.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabd8ff9700 15 client.4259.objecter _session_command_op_remove 5 2 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.370+0000 7fabe1e97700 10 librados: watch_flush enter 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 10 librados: watch_flush exit 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 10 client.4259.objecter close_session for osd.5 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 >> v1:172.21.15.129:6800/21048 conn(0x7fabbc002f70 legacy=0x7fabbc005420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 20 client.4259.objecter put_session s=0x7fabbc002880 osd=5 1 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 20 client.4259.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 10 client.4259.objecter successfully canceled tick 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 >> v1:172.21.15.129:6812/20806 conn(0x7fabc80384e0 legacy=0x7fabc803a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 10 monclient: shutdown 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 >> v1:172.21.15.112:6790/0 conn(0x7fabdc12edc0 legacy=0x7fabdc141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1696700 1 -- 172.21.15.112:0/1547434451 reap_dead start 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 shutdown_connections 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 >> 172.21.15.112:0/1547434451 conn(0x7fabdc067da0 msgr2=0x7fabdc06a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 shutdown_connections 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 1 -- 172.21.15.112:0/1547434451 wait complete. 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.374+0000 7fabe1e97700 1 librados: shutdown 2024-09-18T11:44:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7f09227fc700 10 monclient: get_auth_request con 0x7f090c03f690 auth_method 0 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 15 client.4274.objecter _session_command_op_assign -1 2 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 20 client.4274.objecter _get_session s=0x7fa678002880 osd=1 3 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 20 client.4274.objecter put_session s=0x7fa678002880 osd=1 3 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 20 client.4274.objecter _get_session s=0x7fa678002880 osd=1 3 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 15 client.4274.objecter _session_command_op_remove -1 2 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 20 client.4274.objecter get_session s=0x7fa678002880 osd=1 3 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 15 client.4274.objecter _session_command_op_assign 1 2 2024-09-18T11:44:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 20 client.4274.objecter put_session s=0x7fa678002880 osd=1 4 2024-09-18T11:44:25.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 10 client.4274.objecter _send_command 2 2024-09-18T11:44:25.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 --> v1:172.21.15.112:6800/20985 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6780011c0 con 0x7fa678002f70 2024-09-18T11:44:25.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa68d7fa700 1 -- 172.21.15.112:0/3963738824 <== osd.1 v1:172.21.15.112:6800/20985 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 654699149) 0x7fa6780011c0 con 0x7fa678002f70 2024-09-18T11:44:25.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa68d7fa700 10 client.4274.objecter ms_dispatch 0x7fa698000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T11:44:25.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa68d7fa700 10 client.4274.objecter _finish_command 2 = system:0 2024-09-18T11:44:25.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa68d7fa700 20 client.4274.objecter put_session s=0x7fa678002880 osd=1 4 2024-09-18T11:44:25.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa68d7fa700 15 client.4274.objecter _session_command_op_remove 1 2 2024-09-18T11:44:25.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 10 librados: watch_flush enter 2024-09-18T11:44:25.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 10 librados: watch_flush exit 2024-09-18T11:44:25.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 10 client.4274.objecter close_session for osd.1 2024-09-18T11:44:25.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 >> v1:172.21.15.112:6800/20985 conn(0x7fa678002f70 legacy=0x7fa678005420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 20 client.4274.objecter put_session s=0x7fa678002880 osd=1 1 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 20 client.4274.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 10 client.4274.objecter successfully canceled tick 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 >> v1:172.21.15.129:6812/20806 conn(0x7fa67c038210 legacy=0x7fa67c03a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 10 monclient: shutdown 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 >> v1:172.21.15.112:6790/0 conn(0x7fa69806ea70 legacy=0x7fa698099ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69dc20700 1 -- 172.21.15.112:0/3963738824 reap_dead start 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 shutdown_connections 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 >> 172.21.15.112:0/3963738824 conn(0x7fa6980680d0 msgr2=0x7fa69809c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 shutdown_connections 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 -- 172.21.15.112:0/3963738824 wait complete. 2024-09-18T11:44:25.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.378+0000 7fa69f683700 1 librados: shutdown 2024-09-18T11:44:25.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.382+0000 7f090affd700 10 client.4272.objecter ms_handle_connect 0x7f090c03f690 2024-09-18T11:44:25.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.382+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== osd.4 v1:172.21.15.129:6808/21047 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f090c043330 con 0x7f090c03f690 2024-09-18T11:44:25.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.382+0000 7f090affd700 10 client.4272.objecter ms_dispatch 0x7f0924000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T11:44:25.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.382+0000 7f090affd700 10 client.4272.objecter _finish_command 1 = system:0 2024-09-18T11:44:25.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.382+0000 7f090affd700 20 client.4272.objecter put_session s=0x7f090c03efe0 osd=4 4 2024-09-18T11:44:25.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.382+0000 7f090affd700 15 client.4272.objecter _session_command_op_remove 4 1 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 15 client.4280.objecter _session_command_op_assign -1 2 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 20 client.4280.objecter _get_session s=0x7f9938002880 osd=0 3 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 20 client.4280.objecter put_session s=0x7f9938002880 osd=0 3 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 20 client.4280.objecter _get_session s=0x7f9938002880 osd=0 3 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 15 client.4280.objecter _session_command_op_remove -1 2 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 20 client.4280.objecter get_session s=0x7f9938002880 osd=0 3 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 15 client.4280.objecter _session_command_op_assign 0 2 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 20 client.4280.objecter put_session s=0x7f9938002880 osd=0 4 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 10 client.4280.objecter _send_command 2 2024-09-18T11:44:25.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 --> v1:172.21.15.112:6804/20984 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f99380011c0 con 0x7f9938002f70 2024-09-18T11:44:25.492 INFO:teuthology.orchestra.run.smithi112.stdout:51539607557 2024-09-18T11:44:25.492 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T11:44:25.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f9954ff9700 1 -- 172.21.15.112:0/815747978 <== osd.0 v1:172.21.15.112:6804/20984 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f99380011c0 con 0x7f9938002f70 2024-09-18T11:44:25.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f9954ff9700 10 client.4280.objecter ms_dispatch 0x7f9958000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T11:44:25.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f9954ff9700 10 client.4280.objecter _finish_command 2 = system:0 2024-09-18T11:44:25.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f9954ff9700 20 client.4280.objecter put_session s=0x7f9938002880 osd=0 4 2024-09-18T11:44:25.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f9954ff9700 15 client.4280.objecter _session_command_op_remove 0 2 2024-09-18T11:44:25.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 10 librados: watch_flush enter 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 10 librados: watch_flush exit 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 10 client.4280.objecter close_session for osd.0 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 >> v1:172.21.15.112:6804/20984 conn(0x7f9938002f70 legacy=0x7f9938005420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 20 client.4280.objecter put_session s=0x7f9938002880 osd=0 1 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 20 client.4280.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 10 client.4280.objecter successfully canceled tick 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 >> v1:172.21.15.129:6812/20806 conn(0x7f993c038480 legacy=0x7f993c03a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 10 monclient: shutdown 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 >> v1:172.21.15.112:6790/0 conn(0x7f9958132c60 legacy=0x7f99581313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995d7be700 1 -- 172.21.15.112:0/815747978 reap_dead start 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 shutdown_connections 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 >> 172.21.15.112:0/815747978 conn(0x7f99580680d0 msgr2=0x7f995806b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 shutdown_connections 2024-09-18T11:44:25.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 -- 172.21.15.112:0/815747978 wait complete. 2024-09-18T11:44:25.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f995f221700 1 librados: shutdown 2024-09-18T11:44:25.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 10 client.4266.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-18T11:44:25.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 15 client.4266.objecter _session_command_op_assign -1 2 2024-09-18T11:44:25.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 20 client.4266.objecter _get_session s=0x7f356c03eeb0 osd=3 3 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 20 client.4266.objecter put_session s=0x7f356c03eeb0 osd=3 3 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 20 client.4266.objecter _get_session s=0x7f356c03eeb0 osd=3 3 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 15 client.4266.objecter _session_command_op_remove -1 2 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 20 client.4266.objecter get_session s=0x7f356c03eeb0 osd=3 3 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 15 client.4266.objecter _session_command_op_assign 3 2 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 20 client.4266.objecter put_session s=0x7f356c03eeb0 osd=3 4 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 10 client.4266.objecter _send_command 2 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.386+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 --> v1:172.21.15.129:6804/21049 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f35600011c0 con 0x7f356c03f560 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f35677fe700 1 -- 172.21.15.112:0/3187856659 <== osd.3 v1:172.21.15.129:6804/21049 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3543570501) 0x7f35600011c0 con 0x7f356c03f560 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f35677fe700 10 client.4266.objecter ms_dispatch 0x7f3580000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f35677fe700 10 client.4266.objecter _finish_command 2 = system:0 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f35677fe700 20 client.4266.objecter put_session s=0x7f356c03eeb0 osd=3 4 2024-09-18T11:44:25.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f35677fe700 15 client.4266.objecter _session_command_op_remove 3 2 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 10 librados: watch_flush enter 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 10 librados: watch_flush exit 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 10 client.4266.objecter close_session for osd.3 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 >> v1:172.21.15.129:6804/21049 conn(0x7f356c03f560 legacy=0x7f356c041990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 20 client.4266.objecter put_session s=0x7f356c03eeb0 osd=3 1 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 20 client.4266.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 10 client.4266.objecter successfully canceled tick 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 >> v1:172.21.15.129:6812/20806 conn(0x7f356c038920 legacy=0x7f356c03add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 10 monclient: shutdown 2024-09-18T11:44:25.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 >> v1:172.21.15.112:6789/0 conn(0x7f358011bb90 legacy=0x7f358014eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f357ffff700 1 -- 172.21.15.112:0/3187856659 reap_dead start 2024-09-18T11:44:25.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 shutdown_connections 2024-09-18T11:44:25.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 >> 172.21.15.112:0/3187856659 conn(0x7f35800680d0 msgr2=0x7f358006d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 shutdown_connections 2024-09-18T11:44:25.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 1 -- 172.21.15.112:0/3187856659 wait complete. 2024-09-18T11:44:25.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.390+0000 7f3585c05700 1 librados: shutdown 2024-09-18T11:44:25.501 INFO:teuthology.orchestra.run.smithi112.stdout:47244640260 2024-09-18T11:44:25.501 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:44:25.502 INFO:teuthology.orchestra.run.smithi112.stdout:47244640259 2024-09-18T11:44:25.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:44:25.503 INFO:teuthology.orchestra.run.smithi112.stdout:51539607557 2024-09-18T11:44:25.503 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 10 client.4272.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 15 client.4272.objecter _session_command_op_assign -1 2 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 20 client.4272.objecter _get_session s=0x7f090c03efe0 osd=4 3 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 20 client.4272.objecter put_session s=0x7f090c03efe0 osd=4 3 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 20 client.4272.objecter _get_session s=0x7f090c03efe0 osd=4 3 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 15 client.4272.objecter _session_command_op_remove -1 2 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 20 client.4272.objecter get_session s=0x7f090c03efe0 osd=4 3 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 15 client.4272.objecter _session_command_op_assign 4 2 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 20 client.4272.objecter put_session s=0x7f090c03efe0 osd=4 4 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 10 client.4272.objecter _send_command 2 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 --> v1:172.21.15.129:6808/21047 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f092405f2e0 con 0x7f090c03f690 2024-09-18T11:44:25.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f090affd700 1 -- 172.21.15.112:0/2743684919 <== osd.4 v1:172.21.15.129:6808/21047 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3543570501) 0x7f092405f2e0 con 0x7f090c03f690 2024-09-18T11:44:25.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f090affd700 10 client.4272.objecter ms_dispatch 0x7f0924000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T11:44:25.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f090affd700 10 client.4272.objecter _finish_command 2 = system:0 2024-09-18T11:44:25.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f090affd700 20 client.4272.objecter put_session s=0x7f090c03efe0 osd=4 4 2024-09-18T11:44:25.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f090affd700 15 client.4272.objecter _session_command_op_remove 4 2 2024-09-18T11:44:25.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 10 librados: watch_flush enter 2024-09-18T11:44:25.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 10 librados: watch_flush exit 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 10 client.4272.objecter close_session for osd.4 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 >> v1:172.21.15.129:6808/21047 conn(0x7f090c03f690 legacy=0x7f090c041ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 20 client.4272.objecter put_session s=0x7f090c03efe0 osd=4 1 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 20 client.4272.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 10 client.4272.objecter successfully canceled tick 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 >> v1:172.21.15.129:6812/20806 conn(0x7f090c038b00 legacy=0x7f090c03afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 10 monclient: shutdown 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 >> v1:172.21.15.112:6789/0 conn(0x7f092412e7c0 legacy=0x7f0924145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09237fe700 1 -- 172.21.15.112:0/2743684919 reap_dead start 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 shutdown_connections 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 >> 172.21.15.112:0/2743684919 conn(0x7f09240680d0 msgr2=0x7f092409d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 shutdown_connections 2024-09-18T11:44:25.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 -- 172.21.15.112:0/2743684919 wait complete. 2024-09-18T11:44:25.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.402+0000 7f09293b9700 1 librados: shutdown 2024-09-18T11:44:25.508 INFO:teuthology.orchestra.run.smithi112.stdout:51539607557 2024-09-18T11:44:25.508 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T11:44:25.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient: monmap: 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.646098+0000 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.646098+0000 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 1 Processor -- start 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 1 -- start start 2024-09-18T11:44:25.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient: init 2024-09-18T11:44:25.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:25.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient(hunting): picked mon.noname-a con 0x7fbd940a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient(hunting): picked mon.noname-c con 0x7fbd94097a90 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient(hunting): picked mon.noname-b con 0x7fbd94135220 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9406dee0 con 0x7fbd940a2040 2024-09-18T11:44:25.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd93fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbd940a2040 0x7fbd9406d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35128/0 (socket says 172.21.15.112:35128) 2024-09-18T11:44:25.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd93fff700 1 -- 172.21.15.112:0/3254710262 learned_addr learned my addr 172.21.15.112:0/3254710262 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 1 -- 172.21.15.112:0/3254710262 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9406e110 con 0x7fbd94097a90 2024-09-18T11:44:25.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 1 -- 172.21.15.112:0/3254710262 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd94096dd0 con 0x7fbd94135220 2024-09-18T11:44:25.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.642+0000 7fbd99159700 10 monclient(hunting): authenticate will time out at 837.487853s 2024-09-18T11:44:25.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd93fff700 1 -- 172.21.15.112:0/3254710262 >> v1:172.21.15.112:6789/0 conn(0x7fbd940a2040 legacy=0x7fbd9406d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd98958700 1 -- 172.21.15.112:0/3254710262 >> v1:172.21.15.112:6790/0 conn(0x7fbd94135220 legacy=0x7fbd94137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd937fe700 1 -- 172.21.15.112:0/3254710262 >> v1:172.21.15.129:6789/0 conn(0x7fbd94097a90 legacy=0x7fbd9406d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662991026 0 0) 0x7fbd94096dd0 con 0x7fbd94135220 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient(hunting): my global_id is 4286 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd80003570 con 0x7fbd94135220 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337975067 0 0) 0x7fbd9406dee0 con 0x7fbd940a2040 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient(hunting): my global_id is 4275 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd94096dd0 con 0x7fbd940a2040 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148266232 0 0) 0x7fbd94096dd0 con 0x7fbd940a2040 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 >> v1:172.21.15.112:6790/0 conn(0x7fbd94135220 legacy=0x7fbd94137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 >> v1:172.21.15.129:6789/0 conn(0x7fbd94097a90 legacy=0x7fbd9406d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 monclient: found mon.noname-a 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd94096a60 con 0x7fbd940a2040 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbd84001920 con 0x7fbd940a2040 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd99159700 5 monclient: authenticate success, global_id 4275 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd99159700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: dump: 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.653 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd99159700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd84001bb0 con 0x7fbd940a2040 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 1 -- 172.21.15.112:0/3254710262 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbd84001da0 con 0x7fbd940a2040 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd99159700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd99159700 4 set_mon_vals no callback set 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.646+0000 7fbd92ffd700 10 monclient: dump: 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.654 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.655 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.655 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.655 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.655 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.655 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.655 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient: shutdown 2024-09-18T11:44:25.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- 172.21.15.112:0/3254710262 >> v1:172.21.15.112:6789/0 conn(0x7fbd940a2040 legacy=0x7fbd9406d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- 172.21.15.112:0/3254710262 shutdown_connections 2024-09-18T11:44:25.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- 172.21.15.112:0/3254710262 >> 172.21.15.112:0/3254710262 conn(0x7fbd94067da0 msgr2=0x7fbd940a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- 172.21.15.112:0/3254710262 shutdown_connections 2024-09-18T11:44:25.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- 172.21.15.112:0/3254710262 wait complete. 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient: monmap: 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 librados: starting msgr at 2024-09-18T11:44:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 librados: starting objecter 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 Processor -- start 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- start start 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 librados: setting wanted keys 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 librados: calling monclient init 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient: init 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7ff36d9b9700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7ff36d9b9700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7ff36d9b9700 10 monclient: monmap: 2024-09-18T11:44:25.658 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.653870+0000 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.653870+0000 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient(hunting): picked mon.noname-c con 0x7fbd94097a90 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient(hunting): picked mon.noname-b con 0x7fbd940a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient(hunting): picked mon.noname-a con 0x7fbd94135220 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9406f0f0 con 0x7fbd94135220 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd94139e00 con 0x7fbd940a2040 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd94099b80 con 0x7fbd94097a90 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd99159700 10 monclient(hunting): authenticate will time out at 837.494638s 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd98958700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbd94135220 0x7fbd941476d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35140/0 (socket says 172.21.15.112:35140) 2024-09-18T11:44:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.650+0000 7fbd98958700 1 -- 172.21.15.112:0/4038887935 learned_addr learned my addr 172.21.15.112:0/4038887935 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 1 Processor -- start 2024-09-18T11:44:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 1 -- start start 2024-09-18T11:44:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 10 monclient: init 2024-09-18T11:44:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 10 monclient(hunting): picked mon.noname-c con 0x7ff36809ae70 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 10 monclient(hunting): picked mon.noname-a con 0x7ff36809ffb0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 10 monclient(hunting): picked mon.noname-b con 0x7ff368138ca0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd98958700 1 -- 172.21.15.112:0/4038887935 >> v1:172.21.15.112:6789/0 conn(0x7fbd94135220 legacy=0x7fbd941476d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 client.?.objecter ms_handle_connect 0x7fbd94135220 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 client.?.objecter ms_handle_connect 0x7fbd94097a90 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd93fff700 1 -- 172.21.15.112:0/4038887935 >> v1:172.21.15.112:6790/0 conn(0x7fbd94097a90 legacy=0x7fbd9409d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd937fe700 1 -- 172.21.15.112:0/4038887935 >> v1:172.21.15.129:6789/0 conn(0x7fbd940a2040 legacy=0x7fbd94143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 client.?.objecter ms_handle_connect 0x7fbd940a2040 2024-09-18T11:44:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36806a960 con 0x7ff36809ffb0 2024-09-18T11:44:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36809b240 con 0x7ff36809ae70 2024-09-18T11:44:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3680a16c0 con 0x7ff368138ca0 2024-09-18T11:44:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff36d9b9700 10 monclient(hunting): authenticate will time out at 837.497049s 2024-09-18T11:44:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff3667fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff36809ffb0 0x7ff368137580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35154/0 (socket says 172.21.15.112:35154) 2024-09-18T11:44:25.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff3667fc700 1 -- 172.21.15.112:0/3229132867 learned_addr learned my addr 172.21.15.112:0/3229132867 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198922759 0 0) 0x7fbd9406f0f0 con 0x7fbd94135220 2024-09-18T11:44:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): my global_id is 4278 2024-09-18T11:44:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd74004f10 con 0x7fbd94135220 2024-09-18T11:44:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264643067 0 0) 0x7fbd94099b80 con 0x7fbd94097a90 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): my global_id is 4289 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd9406f0f0 con 0x7fbd94097a90 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892941873 0 0) 0x7fbd94139e00 con 0x7fbd940a2040 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): my global_id is 4282 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd94099b80 con 0x7fbd940a2040 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096957298 0 0) 0x7fbd74004f10 con 0x7fbd94135220 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd94139e00 con 0x7fbd94135220 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2485653437 0 0) 0x7fbd9406f0f0 con 0x7fbd94097a90 2024-09-18T11:44:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd74004f10 con 0x7fbd94097a90 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbd8c002280 con 0x7fbd94135220 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): dump: 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.669 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbd84004830 con 0x7fbd94097a90 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient(hunting): dump: 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2503102227 0 0) 0x7fbd94139e00 con 0x7fbd94135220 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 >> v1:172.21.15.112:6790/0 conn(0x7fbd94097a90 legacy=0x7fbd9409d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 >> v1:172.21.15.129:6789/0 conn(0x7fbd940a2040 legacy=0x7fbd94143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 monclient: found mon.a 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9409b640 con 0x7fbd94135220 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 5 monclient: authenticate success, global_id 4278 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 10 monclient: _renew_subs 2024-09-18T11:44:25.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9409a750 con 0x7fbd94135220 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 10 monclient: _renew_subs 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd940992d0 con 0x7fbd94135220 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd99159700 1 librados: init done 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd8c0027c0 con 0x7fbd94135220 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbd8c002b10 con 0x7fbd94135220 2024-09-18T11:44:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7fbd927fc700 4 set_mon_vals no callback set 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff3667fc700 1 -- 172.21.15.112:0/3229132867 >> v1:172.21.15.112:6789/0 conn(0x7ff36809ffb0 legacy=0x7ff368137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.654+0000 7ff366ffd700 1 -- 172.21.15.112:0/3229132867 >> v1:172.21.15.129:6789/0 conn(0x7ff36809ae70 legacy=0x7ff36806e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338571940 0 0) 0x7ff36809b240 con 0x7ff36809ae70 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient(hunting): my global_id is 4285 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff350003570 con 0x7ff36809ae70 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541407218 0 0) 0x7ff36806a960 con 0x7ff36809ffb0 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient(hunting): my global_id is 4281 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff36809b240 con 0x7ff36809ffb0 2024-09-18T11:44:25.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff3677fe700 1 -- 172.21.15.112:0/3229132867 >> v1:172.21.15.112:6790/0 conn(0x7ff368138ca0 legacy=0x7ff36813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1419062299 0 0) 0x7ff36809b240 con 0x7ff36809ffb0 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 >> v1:172.21.15.112:6790/0 conn(0x7ff368138ca0 legacy=0x7ff36813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 >> v1:172.21.15.129:6789/0 conn(0x7ff36809ae70 legacy=0x7ff36806e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 monclient: found mon.noname-a 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36809bed0 con 0x7ff36809ffb0 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3580018a0 con 0x7ff36809ffb0 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: dump: 2024-09-18T11:44:25.675 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 5 monclient: authenticate success, global_id 4281 2024-09-18T11:44:25.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 10 monclient: dump: 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fbd8c002c70 con 0x7fbd94135220 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff358001b30 con 0x7ff36809ffb0 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 1 -- 172.21.15.112:0/3229132867 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3580048f0 con 0x7ff36809ffb0 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 4 set_mon_vals no callback set 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff365ffb700 10 monclient: dump: 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.678 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.679 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.679 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.679 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.679 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.679 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.679 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.679 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7fbd8c043b30 con 0x7fbd94135220 2024-09-18T11:44:25.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 10 client.4278.objecter ms_dispatch 0x7fbd94000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 3 client.4278.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 3 client.4278.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd8bfff700 20 client.4278.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd93fff700 10 monclient: get_auth_request con 0x7fbd740382c0 auth_method 0 2024-09-18T11:44:25.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 10 monclient: shutdown 2024-09-18T11:44:25.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 -- 172.21.15.112:0/3229132867 >> v1:172.21.15.112:6789/0 conn(0x7ff36809ffb0 legacy=0x7ff368137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 -- 172.21.15.112:0/3229132867 shutdown_connections 2024-09-18T11:44:25.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 -- 172.21.15.112:0/3229132867 >> 172.21.15.112:0/3229132867 conn(0x7ff368068110 msgr2=0x7ff3680a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 -- 172.21.15.112:0/3229132867 shutdown_connections 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 -- 172.21.15.112:0/3229132867 wait complete. 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 10 monclient: monmap: 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.682 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 librados: starting msgr at 2024-09-18T11:44:25.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 librados: starting objecter 2024-09-18T11:44:25.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 Processor -- start 2024-09-18T11:44:25.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 -- start start 2024-09-18T11:44:25.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 librados: setting wanted keys 2024-09-18T11:44:25.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 1 librados: calling monclient init 2024-09-18T11:44:25.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7ff36d9b9700 10 monclient: init 2024-09-18T11:44:25.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd99159700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd99159700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd99159700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.658+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd780050b0 con 0x7fbd94135220 2024-09-18T11:44:25.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:25.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7fbd8bfff700 10 client.4278.objecter ms_handle_connect 0x7fbd740382c0 2024-09-18T11:44:25.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 10 monclient(hunting): picked mon.noname-a con 0x7ff36809ae70 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 10 monclient(hunting): picked mon.noname-b con 0x7ff36809ffb0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 10 monclient(hunting): picked mon.noname-c con 0x7ff368138ca0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36806b7a0 con 0x7ff36809ae70 2024-09-18T11:44:25.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3681231e0 con 0x7ff36809ffb0 2024-09-18T11:44:25.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36811fcd0 con 0x7ff368138ca0 2024-09-18T11:44:25.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff36d9b9700 10 monclient(hunting): authenticate will time out at 837.504657s 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff366ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff36809ae70 0x7ff36811eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35168/0 (socket says 172.21.15.112:35168) 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff366ffd700 1 -- 172.21.15.112:0/98841267 learned_addr learned my addr 172.21.15.112:0/98841267 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff366ffd700 1 -- 172.21.15.112:0/98841267 >> v1:172.21.15.112:6789/0 conn(0x7ff36809ae70 legacy=0x7ff36811eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 client.?.objecter ms_handle_connect 0x7ff36809ae70 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 client.?.objecter ms_handle_connect 0x7ff368138ca0 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff3677fe700 1 -- 172.21.15.112:0/98841267 >> v1:172.21.15.112:6790/0 conn(0x7ff368138ca0 legacy=0x7ff36814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420823668 0 0) 0x7ff36811fcd0 con 0x7ff368138ca0 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): my global_id is 4295 2024-09-18T11:44:25.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff354004f10 con 0x7ff368138ca0 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 client.?.objecter ms_handle_connect 0x7ff36809ffb0 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff3667fc700 1 -- 172.21.15.112:0/98841267 >> v1:172.21.15.129:6789/0 conn(0x7ff36809ffb0 legacy=0x7ff36811f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1121485023 0 0) 0x7ff354004f10 con 0x7ff368138ca0 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff36811fcd0 con 0x7ff368138ca0 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff35c001b80 con 0x7ff368138ca0 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): dump: 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.689 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054487275 0 0) 0x7ff3681231e0 con 0x7ff36809ffb0 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 10 monclient(hunting): my global_id is 4288 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.662+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff354004f10 con 0x7ff36809ffb0 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 447439838 0 0) 0x7ff36811fcd0 con 0x7ff368138ca0 2024-09-18T11:44:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 >> v1:172.21.15.129:6789/0 conn(0x7ff36809ffb0 legacy=0x7ff36811f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 >> v1:172.21.15.112:6789/0 conn(0x7ff36809ae70 legacy=0x7ff36811eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 monclient: found mon.c 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368120f00 con 0x7ff368138ca0 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 5 monclient: authenticate success, global_id 4295 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 10 monclient: _renew_subs 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3681209f0 con 0x7ff368138ca0 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 10 monclient: _renew_subs 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff368137a90 con 0x7ff368138ca0 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff36d9b9700 1 librados: init done 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff35c0022b0 con 0x7ff368138ca0 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff35c002600 con 0x7ff368138ca0 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff364ff9700 4 set_mon_vals no callback set 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 10 monclient: dump: 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.692 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fbd8c043e60 con 0x7fbd94135220 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7fbd8bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7fbd8bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ff35c002860 con 0x7ff368138ca0 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7ff35c0436c0 con 0x7ff368138ca0 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 10 client.4295.objecter ms_dispatch 0x7ff368000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 3 client.4295.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 3 client.4295.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff34effd700 20 client.4295.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.666+0000 7ff366ffd700 10 monclient: get_auth_request con 0x7ff3540382a0 auth_method 0 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.670+0000 7ff36d9b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.670+0000 7ff36d9b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.670+0000 7ff36d9b9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.670+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3480050b0 con 0x7ff368138ca0 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.670+0000 7ff34effd700 10 client.4295.objecter ms_handle_connect 0x7ff3540382a0 2024-09-18T11:44:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.674+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7ff35c002b10 con 0x7ff368138ca0 2024-09-18T11:44:25.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.674+0000 7ff34effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.674+0000 7ff34effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:25.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient: monmap: 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.680483+0000 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.680483+0000 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.697 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient: monmap: 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.680607+0000 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.680607+0000 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.698 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient: monmap: 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.680825+0000 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.680825+0000 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.699 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.700 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.700 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.700 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.700 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 1 Processor -- start 2024-09-18T11:44:25.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 1 -- start start 2024-09-18T11:44:25.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient: init 2024-09-18T11:44:25.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 1 Processor -- start 2024-09-18T11:44:25.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 1 -- start start 2024-09-18T11:44:25.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient: init 2024-09-18T11:44:25.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient(hunting): picked mon.noname-b con 0x7f06e006a8b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 1 Processor -- start 2024-09-18T11:44:25.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 1 -- start start 2024-09-18T11:44:25.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient: init 2024-09-18T11:44:25.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient: get_monmap_and_config 2024-09-18T11:44:25.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient: monmap: 2024-09-18T11:44:25.702 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.702 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:25.682215+0000 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:25.682215+0000 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient(hunting): picked mon.noname-c con 0x7f06e0132330 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient(hunting): picked mon.noname-a con 0x7f06e0132e40 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e0096ad0 con 0x7f06e0132e40 2024-09-18T11:44:25.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e00a20e0 con 0x7f06e0132330 2024-09-18T11:44:25.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e00a1740 con 0x7f06e006a8b0 2024-09-18T11:44:25.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e6263700 10 monclient(hunting): authenticate will time out at 837.521891s 2024-09-18T11:44:25.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e4800700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f06e0132e40 0x7f06e013a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35182/0 (socket says 172.21.15.112:35182) 2024-09-18T11:44:25.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f06e4800700 1 -- 172.21.15.112:0/1650188975 learned_addr learned my addr 172.21.15.112:0/1650188975 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:25.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient(hunting): picked mon.noname-c con 0x7efdb812f4a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient(hunting): picked mon.noname-b con 0x7efdb806e620 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient(hunting): picked mon.noname-a con 0x7efdb8135640 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb809f200 con 0x7efdb8135640 2024-09-18T11:44:25.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb806f100 con 0x7efdb812f4a0 2024-09-18T11:44:25.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb809eb30 con 0x7efdb806e620 2024-09-18T11:44:25.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdbd19d700 10 monclient(hunting): authenticate will time out at 837.521927s 2024-09-18T11:44:25.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdb659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7efdb806e620 0x7efdb806e9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41518/0 (socket says 172.21.15.112:41518) 2024-09-18T11:44:25.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7efdb659c700 1 -- 172.21.15.112:0/2090818259 learned_addr learned my addr 172.21.15.112:0/2090818259 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient(hunting): picked mon.noname-c con 0x7f045c09f270 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient(hunting): picked mon.noname-a con 0x7f045c06eb40 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient(hunting): picked mon.noname-b con 0x7f045c06ef10 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c06b030 con 0x7f045c06eb40 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c09f640 con 0x7f045c09f270 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c137340 con 0x7f045c06ef10 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 1 Processor -- start 2024-09-18T11:44:25.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f0463236700 10 monclient(hunting): authenticate will time out at 837.522627s 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 1 -- start start 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient: init 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient(hunting): picked mon.noname-b con 0x7f65580a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient(hunting): picked mon.noname-c con 0x7f655806e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient(hunting): picked mon.noname-a con 0x7f6558135300 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65580a1600 con 0x7f6558135300 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655806eda0 con 0x7f655806e9d0 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655806b5a0 con 0x7f65580a1230 2024-09-18T11:44:25.709 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.678+0000 7f655ff2f700 10 monclient(hunting): authenticate will time out at 837.523347s 2024-09-18T11:44:25.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06dffff700 1 -- 172.21.15.112:0/1650188975 >> v1:172.21.15.112:6790/0 conn(0x7f06e006a8b0 legacy=0x7f06e0097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06e4800700 1 -- 172.21.15.112:0/1650188975 >> v1:172.21.15.112:6789/0 conn(0x7f06e0132e40 legacy=0x7f06e013a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f04617d3700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f045c06ef10 0x7f045c13ef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41520/0 (socket says 172.21.15.112:41520) 2024-09-18T11:44:25.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f04617d3700 1 -- 172.21.15.112:0/3990204950 learned_addr learned my addr 172.21.15.112:0/3990204950 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045bfff700 1 --1- 172.21.15.112:0/3990204950 >> v1:172.21.15.112:6789/0 conn(0x7f045c06eb40 0x7f045c13b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35200/0 (socket says 172.21.15.112:35200) 2024-09-18T11:44:25.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655e4cc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6558135300 0x7f65581376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35216/0 (socket says 172.21.15.112:35216) 2024-09-18T11:44:25.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655e4cc700 1 -- 172.21.15.112:0/1698653568 learned_addr learned my addr 172.21.15.112:0/1698653568 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06df7fe700 1 -- 172.21.15.112:0/1650188975 >> v1:172.21.15.129:6789/0 conn(0x7f06e0132330 legacy=0x7f06e0132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb659c700 1 -- 172.21.15.112:0/2090818259 >> v1:172.21.15.112:6790/0 conn(0x7efdb806e620 legacy=0x7efdb806e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541293798 0 0) 0x7f06e0096ad0 con 0x7f06e0132e40 2024-09-18T11:44:25.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): my global_id is 4284 2024-09-18T11:44:25.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06c8003570 con 0x7f06e0132e40 2024-09-18T11:44:25.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562187440 0 0) 0x7efdb809eb30 con 0x7efdb806e620 2024-09-18T11:44:25.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient(hunting): my global_id is 4301 2024-09-18T11:44:25.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda0003570 con 0x7efdb806e620 2024-09-18T11:44:25.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135281742 0 0) 0x7efda0003570 con 0x7efdb806e620 2024-09-18T11:44:25.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 >> v1:172.21.15.129:6789/0 conn(0x7efdb812f4a0 legacy=0x7efdb812f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T11:44:25.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 >> v1:172.21.15.112:6789/0 conn(0x7efdb8135640 legacy=0x7efdb8137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 monclient: found mon.noname-b 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb806c850 con 0x7efdb806e620 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 5 monclient: authenticate success, global_id 4301 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efda8001a00 con 0x7efdb806e620 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: dump: 2024-09-18T11:44:25.715 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efda8001c60 con 0x7efdb806e620 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 1 -- 172.21.15.112:0/2090818259 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efda8004a90 con 0x7efdb806e620 2024-09-18T11:44:25.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 4 set_mon_vals no callback set 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdb5d9b700 10 monclient: dump: 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.717 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 10 monclient: shutdown 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 1 -- 172.21.15.112:0/2090818259 >> v1:172.21.15.112:6790/0 conn(0x7efdb806e620 legacy=0x7efdb806e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 1 -- 172.21.15.112:0/2090818259 shutdown_connections 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 1 -- 172.21.15.112:0/2090818259 >> 172.21.15.112:0/2090818259 conn(0x7efdb806a230 msgr2=0x7efdb80a2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 1 -- 172.21.15.112:0/2090818259 shutdown_connections 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 1 -- 172.21.15.112:0/2090818259 wait complete. 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924416828 0 0) 0x7f06e00a1740 con 0x7f06e006a8b0 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): my global_id is 4298 2024-09-18T11:44:25.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e0096ad0 con 0x7f06e006a8b0 2024-09-18T11:44:25.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045bfff700 1 -- 172.21.15.112:0/3990204950 >> v1:172.21.15.112:6789/0 conn(0x7f045c06eb40 legacy=0x7f045c13b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f04617d3700 1 -- 172.21.15.112:0/3990204950 >> v1:172.21.15.112:6790/0 conn(0x7f045c06ef10 legacy=0x7f045c13ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655dccb700 1 -- 172.21.15.112:0/1698653568 >> v1:172.21.15.112:6790/0 conn(0x7f65580a1230 legacy=0x7f655806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f0460fd2700 1 -- 172.21.15.112:0/3990204950 >> v1:172.21.15.129:6789/0 conn(0x7f045c09f270 legacy=0x7f045c1364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320915955 0 0) 0x7f06e00a20e0 con 0x7f06e0132330 2024-09-18T11:44:25.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient(hunting): my global_id is 4291 2024-09-18T11:44:25.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e00a1740 con 0x7f06e0132330 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387159951 0 0) 0x7f045c06b030 con 0x7f045c06eb40 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient(hunting): my global_id is 4287 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0444003570 con 0x7f045c06eb40 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848697982 0 0) 0x7f045c137340 con 0x7f045c06ef10 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient(hunting): my global_id is 4304 2024-09-18T11:44:25.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045c06b030 con 0x7f045c06ef10 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655d4ca700 1 -- 172.21.15.112:0/1698653568 >> v1:172.21.15.129:6789/0 conn(0x7f655806e9d0 legacy=0x7f655806ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 668712636 0 0) 0x7f06c8003570 con 0x7f06e0132e40 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 >> v1:172.21.15.112:6790/0 conn(0x7f06e006a8b0 legacy=0x7f06e0097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 >> v1:172.21.15.129:6789/0 conn(0x7f06e0132330 legacy=0x7f06e0132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 monclient: found mon.noname-a 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e013ece0 con 0x7f06e0132e40 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f06d4001870 con 0x7f06e0132e40 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: dump: 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.723 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.724 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.724 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655e4cc700 1 -- 172.21.15.112:0/1698653568 >> v1:172.21.15.112:6789/0 conn(0x7f6558135300 legacy=0x7f65581376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164086288 0 0) 0x7f655806eda0 con 0x7f655806e9d0 2024-09-18T11:44:25.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06e6263700 5 monclient: authenticate success, global_id 4284 2024-09-18T11:44:25.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06e6263700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 10 monclient: monmap: 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.725 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06d4001b00 con 0x7f06e0132e40 2024-09-18T11:44:25.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 1 -- 172.21.15.112:0/1650188975 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f06d40048f0 con 0x7f06e0132e40 2024-09-18T11:44:25.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06e6263700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06e6263700 4 set_mon_vals no callback set 2024-09-18T11:44:25.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 759101313 0 0) 0x7f0444003570 con 0x7f045c06eb40 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 >> v1:172.21.15.112:6790/0 conn(0x7f045c06ef10 legacy=0x7f045c13ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 >> v1:172.21.15.129:6789/0 conn(0x7f045c09f270 legacy=0x7f045c1364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 monclient: found mon.noname-a 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c06f320 con 0x7f045c06eb40 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f044c001940 con 0x7f045c06eb40 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: dump: 2024-09-18T11:44:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f0463236700 5 monclient: authenticate success, global_id 4287 2024-09-18T11:44:25.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f0463236700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f06deffd700 10 monclient: dump: 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.729 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f044c001bd0 con 0x7f045c06eb40 2024-09-18T11:44:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 1 -- 172.21.15.112:0/3990204950 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f044c0049d0 con 0x7f045c06eb40 2024-09-18T11:44:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f045b7fe700 10 monclient: dump: 2024-09-18T11:44:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f0463236700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f0463236700 4 set_mon_vals no callback set 2024-09-18T11:44:25.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): my global_id is 4294 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6544003570 con 0x7f655806e9d0 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775922497 0 0) 0x7f655806b5a0 con 0x7f65580a1230 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): my global_id is 4307 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f655806eda0 con 0x7f65580a1230 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808485006 0 0) 0x7f65580a1600 con 0x7f6558135300 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient(hunting): my global_id is 4290 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f655806b5a0 con 0x7f6558135300 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519700922 0 0) 0x7f6544003570 con 0x7f655806e9d0 2024-09-18T11:44:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 >> v1:172.21.15.112:6790/0 conn(0x7f65580a1230 legacy=0x7f655806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 >> v1:172.21.15.112:6789/0 conn(0x7f6558135300 legacy=0x7f65581376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 monclient: found mon.noname-c 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655809f9e0 con 0x7f655806e9d0 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6550001d10 con 0x7f655806e9d0 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ccc9700 10 monclient: dump: 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ff2f700 5 monclient: authenticate success, global_id 4294 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7f655ff2f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6550001fa0 con 0x7f655806e9d0 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ccc9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ccc9700 1 -- 172.21.15.112:0/1698653568 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6550002420 con 0x7f655806e9d0 2024-09-18T11:44:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 4 set_mon_vals no callback set 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ccc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ccc9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ccc9700 10 monclient: dump: 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.736 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient: shutdown 2024-09-18T11:44:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- 172.21.15.112:0/1650188975 >> v1:172.21.15.112:6789/0 conn(0x7f06e0132e40 legacy=0x7f06e013a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 1 librados: starting msgr at 2024-09-18T11:44:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- 172.21.15.112:0/1650188975 shutdown_connections 2024-09-18T11:44:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- 172.21.15.112:0/1650188975 >> 172.21.15.112:0/1650188975 conn(0x7f06e00a5b70 msgr2=0x7f06e00a5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 10 monclient: shutdown 2024-09-18T11:44:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 1 -- 172.21.15.112:0/3990204950 >> v1:172.21.15.112:6789/0 conn(0x7f045c06eb40 legacy=0x7f045c13b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.682+0000 7efdbd19d700 1 librados: starting objecter 2024-09-18T11:44:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 1 Processor -- start 2024-09-18T11:44:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 1 -- start start 2024-09-18T11:44:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 1 librados: setting wanted keys 2024-09-18T11:44:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 1 librados: calling monclient init 2024-09-18T11:44:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 10 monclient: init 2024-09-18T11:44:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 10 monclient(hunting): picked mon.noname-c con 0x7efdb806e620 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 10 monclient(hunting): picked mon.noname-b con 0x7efdb812f4a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 10 monclient(hunting): picked mon.noname-a con 0x7efdb8135640 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb809a230 con 0x7efdb8135640 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb8098b20 con 0x7efdb812f4a0 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb809ccf0 con 0x7efdb806e620 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdbd19d700 10 monclient(hunting): authenticate will time out at 837.528925s 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdb759e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7efdb8135640 0x7efdb814e450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35218/0 (socket says 172.21.15.112:35218) 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdb759e700 1 -- 172.21.15.112:0/4243722891 learned_addr learned my addr 172.21.15.112:0/4243722891 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 1 -- 172.21.15.112:0/3990204950 shutdown_connections 2024-09-18T11:44:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 1 -- 172.21.15.112:0/3990204950 >> 172.21.15.112:0/3990204950 conn(0x7f045c0a5b70 msgr2=0x7f045c0a5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- 172.21.15.112:0/1650188975 shutdown_connections 2024-09-18T11:44:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- 172.21.15.112:0/1650188975 wait complete. 2024-09-18T11:44:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 1 -- 172.21.15.112:0/3990204950 shutdown_connections 2024-09-18T11:44:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient: shutdown 2024-09-18T11:44:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 -- 172.21.15.112:0/1698653568 >> v1:172.21.15.129:6789/0 conn(0x7f655806e9d0 legacy=0x7f655806ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 -- 172.21.15.112:0/1698653568 shutdown_connections 2024-09-18T11:44:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 -- 172.21.15.112:0/1698653568 >> 172.21.15.112:0/1698653568 conn(0x7f65580680d0 msgr2=0x7f65580a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 -- 172.21.15.112:0/1698653568 shutdown_connections 2024-09-18T11:44:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 -- 172.21.15.112:0/1698653568 wait complete. 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient: monmap: 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.744 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient: monmap: 2024-09-18T11:44:25.744 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.744 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.744 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.744 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.744 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.745 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.745 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.745 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.745 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.745 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 librados: starting msgr at 2024-09-18T11:44:25.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 librados: starting objecter 2024-09-18T11:44:25.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 Processor -- start 2024-09-18T11:44:25.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- start start 2024-09-18T11:44:25.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 librados: setting wanted keys 2024-09-18T11:44:25.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 librados: calling monclient init 2024-09-18T11:44:25.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient: init 2024-09-18T11:44:25.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdb759e700 1 -- 172.21.15.112:0/4243722891 >> v1:172.21.15.112:6789/0 conn(0x7efdb8135640 legacy=0x7efdb814e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient(hunting): picked mon.noname-c con 0x7f06e006a8b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient(hunting): picked mon.noname-b con 0x7f06e0132330 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient(hunting): picked mon.noname-a con 0x7f06e0132e40 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e00a4380 con 0x7f06e0132e40 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e01353d0 con 0x7f06e0132330 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e01350f0 con 0x7f06e006a8b0 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06e6263700 10 monclient(hunting): authenticate will time out at 837.531428s 2024-09-18T11:44:25.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdb759e700 1 --1- 172.21.15.112:0/4243722891 >> v1:172.21.15.112:6789/0 conn(0x7efdb8135640 0x7efdb814e450 :-1 s=OPENED pgs=93 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7efdb809a230 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:44:25.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdb659c700 1 -- 172.21.15.112:0/4243722891 >> v1:172.21.15.129:6789/0 conn(0x7efdb812f4a0 legacy=0x7efdb8098410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7efdb6d9d700 1 -- 172.21.15.112:0/4243722891 >> v1:172.21.15.112:6790/0 conn(0x7efdb806e620 legacy=0x7efdb8097d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06dffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f06e006a8b0 0x7f06e009dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41554/0 (socket says 172.21.15.112:41554) 2024-09-18T11:44:25.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f06dffff700 1 -- 172.21.15.112:0/3872958422 learned_addr learned my addr 172.21.15.112:0/3872958422 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 librados: starting msgr at 2024-09-18T11:44:25.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 librados: starting objecter 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 Processor -- start 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 -- start start 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 librados: setting wanted keys 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 librados: calling monclient init 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient: init 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 1 -- 172.21.15.112:0/3990204950 wait complete. 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 10 monclient: build_initial_monmap 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 10 monclient: monmap: 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:25.749 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 1 librados: starting msgr at 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f0463236700 1 librados: starting objecter 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 1 Processor -- start 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 1 -- start start 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 1 librados: setting wanted keys 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 1 librados: calling monclient init 2024-09-18T11:44:25.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 10 monclient: init 2024-09-18T11:44:25.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:25.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient(hunting): picked mon.noname-b con 0x7f655806e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient(hunting): picked mon.noname-c con 0x7f65580a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 10 monclient(hunting): picked mon.noname-a con 0x7f6558135300 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655806ef90 con 0x7f6558135300 2024-09-18T11:44:25.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.686+0000 7f655ff2f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655809d0b0 con 0x7f655806e9d0 2024-09-18T11:44:25.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f655ff2f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655811cde0 con 0x7f65580a1230 2024-09-18T11:44:25.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f655ff2f700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f655ff2f700 10 monclient(hunting): authenticate will time out at 837.532042s 2024-09-18T11:44:25.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f655d4ca700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f65580a1230 0x7f6558142990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41568/0 (socket says 172.21.15.112:41568) 2024-09-18T11:44:25.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f655d4ca700 1 -- 172.21.15.112:0/2603400776 learned_addr learned my addr 172.21.15.112:0/2603400776 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:25.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:25.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 10 monclient(hunting): picked mon.noname-c con 0x7f045c06eb40 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:25.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 10 monclient(hunting): picked mon.noname-b con 0x7f045c06ef10 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:25.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 10 monclient(hunting): picked mon.noname-a con 0x7f045c09f270 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:25.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c121330 con 0x7f045c09f270 2024-09-18T11:44:25.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c0a22e0 con 0x7f045c06ef10 2024-09-18T11:44:25.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c136610 con 0x7f045c06eb40 2024-09-18T11:44:25.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:25.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f0463236700 10 monclient(hunting): authenticate will time out at 837.532825s 2024-09-18T11:44:25.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06dffff700 1 -- 172.21.15.112:0/3872958422 >> v1:172.21.15.112:6790/0 conn(0x7f06e006a8b0 legacy=0x7f06e009dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 client.?.objecter ms_handle_connect 0x7f06e006a8b0 2024-09-18T11:44:25.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 client.?.objecter ms_handle_connect 0x7f06e0132e40 2024-09-18T11:44:25.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e4800700 1 -- 172.21.15.112:0/3872958422 >> v1:172.21.15.112:6789/0 conn(0x7f06e0132e40 legacy=0x7f06e009e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f04617d3700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f045c09f270 0x7f045c14de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35256/0 (socket says 172.21.15.112:35256) 2024-09-18T11:44:25.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f04617d3700 1 -- 172.21.15.112:0/3440591962 learned_addr learned my addr 172.21.15.112:0/3440591962 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:25.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331973762 0 0) 0x7f06e01350f0 con 0x7f06e006a8b0 2024-09-18T11:44:25.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06df7fe700 1 -- 172.21.15.112:0/3872958422 >> v1:172.21.15.129:6789/0 conn(0x7f06e0132330 legacy=0x7f06e009e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient(hunting): my global_id is 4313 2024-09-18T11:44:25.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06c8003570 con 0x7f06e006a8b0 2024-09-18T11:44:25.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 client.?.objecter ms_handle_connect 0x7f06e0132330 2024-09-18T11:44:25.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f655d4ca700 1 -- 172.21.15.112:0/2603400776 >> v1:172.21.15.112:6790/0 conn(0x7f65580a1230 legacy=0x7f6558142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3595123033 0 0) 0x7f06c8003570 con 0x7f06e006a8b0 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06e01350f0 con 0x7f06e006a8b0 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f06d8002280 con 0x7f06e006a8b0 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient(hunting): dump: 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.758 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 256564553 0 0) 0x7f06e01350f0 con 0x7f06e006a8b0 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 >> v1:172.21.15.129:6789/0 conn(0x7f06e0132330 legacy=0x7f06e009e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 >> v1:172.21.15.112:6789/0 conn(0x7f06e0132e40 legacy=0x7f06e009e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 monclient: found mon.c 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e01335e0 con 0x7f06e006a8b0 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 5 monclient: authenticate success, global_id 4313 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 10 monclient: _renew_subs 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e013b070 con 0x7f06e006a8b0 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 10 monclient: _renew_subs 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e0134330 con 0x7f06e006a8b0 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06e6263700 1 librados: init done 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06d8002790 con 0x7f06e006a8b0 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f06d8002ae0 con 0x7f06e006a8b0 2024-09-18T11:44:25.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06de7fc700 4 set_mon_vals no callback set 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 10 monclient: dump: 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.690+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f06d8002cf0 con 0x7f06e006a8b0 2024-09-18T11:44:25.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 client.?.objecter ms_handle_connect 0x7efdb8135640 2024-09-18T11:44:25.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 client.?.objecter ms_handle_connect 0x7efdb812f4a0 2024-09-18T11:44:25.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 client.?.objecter ms_handle_connect 0x7f65580a1230 2024-09-18T11:44:25.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 client.?.objecter ms_handle_connect 0x7efdb806e620 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209031835 0 0) 0x7efdb809ccf0 con 0x7efdb806e620 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): my global_id is 4310 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886132821 0 0) 0x7f655811cde0 con 0x7f65580a1230 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): my global_id is 4316 2024-09-18T11:44:25.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6544003570 con 0x7f65580a1230 2024-09-18T11:44:25.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda4004f10 con 0x7efdb806e620 2024-09-18T11:44:25.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841773406 0 0) 0x7efdb8098b20 con 0x7efdb812f4a0 2024-09-18T11:44:25.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): my global_id is 4300 2024-09-18T11:44:25.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb809ccf0 con 0x7efdb812f4a0 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655dccb700 1 -- 172.21.15.112:0/2603400776 >> v1:172.21.15.129:6789/0 conn(0x7f655806e9d0 legacy=0x7f655813dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 client.?.objecter ms_handle_connect 0x7f655806e9d0 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1453741158 0 0) 0x7f6544003570 con 0x7f65580a1230 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f655811cde0 con 0x7f65580a1230 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f06d8043cf0 con 0x7f06e006a8b0 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f06c7fff700 10 client.4313.objecter ms_dispatch 0x7f06e0000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f06c7fff700 3 client.4313.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f06c7fff700 3 client.4313.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f06c7fff700 20 client.4313.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 client.?.objecter ms_handle_connect 0x7f6558135300 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655e4cc700 1 -- 172.21.15.112:0/2603400776 >> v1:172.21.15.112:6789/0 conn(0x7f6558135300 legacy=0x7f65581460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6550002420 con 0x7f65580a1230 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): dump: 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.766 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.767 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.767 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.767 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.767 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.767 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.767 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.767 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f04617d3700 1 -- 172.21.15.112:0/3440591962 >> v1:172.21.15.112:6789/0 conn(0x7f045c09f270 legacy=0x7f045c14de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 client.?.objecter ms_handle_connect 0x7f045c09f270 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 client.?.objecter ms_handle_connect 0x7f045c06ef10 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f045bfff700 1 -- 172.21.15.112:0/3440591962 >> v1:172.21.15.129:6789/0 conn(0x7f045c06ef10 legacy=0x7f045c135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 client.?.objecter ms_handle_connect 0x7f045c06eb40 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:25.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0460fd2700 1 -- 172.21.15.112:0/3440591962 >> v1:172.21.15.112:6790/0 conn(0x7f045c06eb40 legacy=0x7f045c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:25.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514214495 0 0) 0x7f655806ef90 con 0x7f6558135300 2024-09-18T11:44:25.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient(hunting): my global_id is 4302 2024-09-18T11:44:25.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6544003570 con 0x7f6558135300 2024-09-18T11:44:25.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f06df7fe700 10 monclient: get_auth_request con 0x7f06c8038410 auth_method 0 2024-09-18T11:44:25.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2732231862 0 0) 0x7efda4004f10 con 0x7efdb806e620 2024-09-18T11:44:25.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdb8098b20 con 0x7efdb806e620 2024-09-18T11:44:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 791949700 0 0) 0x7f655811cde0 con 0x7f65580a1230 2024-09-18T11:44:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 >> v1:172.21.15.129:6789/0 conn(0x7f655806e9d0 legacy=0x7f655813dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efdac002aa0 con 0x7efdb806e620 2024-09-18T11:44:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): dump: 2024-09-18T11:44:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.770 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44509832 0 0) 0x7f045c121330 con 0x7f045c09f270 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): my global_id is 4299 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448004f10 con 0x7f045c09f270 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789954482 0 0) 0x7f045c136610 con 0x7f045c06eb40 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): my global_id is 4319 2024-09-18T11:44:25.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045c121330 con 0x7f045c06eb40 2024-09-18T11:44:25.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 >> v1:172.21.15.112:6789/0 conn(0x7f6558135300 legacy=0x7f65581460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 monclient: found mon.c 2024-09-18T11:44:25.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655811c8d0 con 0x7f65580a1230 2024-09-18T11:44:25.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1968960520 0 0) 0x7efdb809ccf0 con 0x7efdb812f4a0 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda4004f10 con 0x7efdb812f4a0 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efda8004830 con 0x7efdb812f4a0 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient(hunting): dump: 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.774 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1384961636 0 0) 0x7efdb8098b20 con 0x7efdb806e620 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 >> v1:172.21.15.129:6789/0 conn(0x7efdb812f4a0 legacy=0x7efdb8098410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 5 monclient: authenticate success, global_id 4316 2024-09-18T11:44:25.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 10 monclient: _renew_subs 2024-09-18T11:44:25.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 >> v1:172.21.15.112:6789/0 conn(0x7efdb8135640 legacy=0x7efdb814e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 1 -- 172.21.15.112:0/2603400776 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655813e450 con 0x7f65580a1230 2024-09-18T11:44:25.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756910868 0 0) 0x7f045c0a22e0 con 0x7f045c06ef10 2024-09-18T11:44:25.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:25.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:25.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): my global_id is 4309 2024-09-18T11:44:25.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045c136610 con 0x7f045c06ef10 2024-09-18T11:44:25.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 10 monclient: _renew_subs 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034364699 0 0) 0x7f0448004f10 con 0x7f045c09f270 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f045c0a22e0 con 0x7f045c09f270 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0454001e70 con 0x7f045c09f270 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): dump: 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.778 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65500029d0 con 0x7f65580a1230 2024-09-18T11:44:25.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 1 -- 172.21.15.112:0/2603400776 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6558134a40 con 0x7f65580a1230 2024-09-18T11:44:25.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655ff2f700 1 librados: init done 2024-09-18T11:44:25.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 monclient: found mon.c 2024-09-18T11:44:25.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb809d3f0 con 0x7efdb806e620 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6550002d60 con 0x7f65580a1230 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 monclient: dump: 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.781 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654ffff700 4 set_mon_vals no callback set 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f6550002fe0 con 0x7f65580a1230 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 5 monclient: authenticate success, global_id 4310 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 10 monclient: _renew_subs 2024-09-18T11:44:25.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb8099830 con 0x7efdb806e620 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 10 monclient: _renew_subs 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb809a400 con 0x7efdb806e620 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdbd19d700 1 librados: init done 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3380376583 0 0) 0x7f045c136610 con 0x7f045c06ef10 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0448004f10 con 0x7f045c06ef10 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 994803808 0 0) 0x7f045c0a22e0 con 0x7f045c09f270 2024-09-18T11:44:25.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 >> v1:172.21.15.112:6790/0 conn(0x7f045c06eb40 legacy=0x7f045c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 >> v1:172.21.15.129:6789/0 conn(0x7f045c06ef10 legacy=0x7f045c135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 monclient: found mon.a 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c122870 con 0x7f045c09f270 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient: _finish_auth 0 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 5 monclient: authenticate success, global_id 4299 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 10 monclient: _renew_subs 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045c121980 con 0x7f045c09f270 2024-09-18T11:44:25.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:25.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 10 monclient: _renew_subs 2024-09-18T11:44:25.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f045c135e60 con 0x7f045c09f270 2024-09-18T11:44:25.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0463236700 1 librados: init done 2024-09-18T11:44:25.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f6550043e30 con 0x7f65580a1230 2024-09-18T11:44:25.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 10 client.4316.objecter ms_dispatch 0x7f6558000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 3 client.4316.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 3 client.4316.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdac001180 con 0x7efdb806e620 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efdac002dd0 con 0x7efdb806e620 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efd9effd700 10 monclient: dump: 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.786 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7efdb559a700 4 set_mon_vals no callback set 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7efdac0014d0 con 0x7efdb806e620 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7efdac043560 con 0x7efdb806e620 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7efd9effd700 10 client.4310.objecter ms_dispatch 0x7efdb8000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7efd9effd700 3 client.4310.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7efd9effd700 3 client.4310.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f06c7fff700 10 client.4313.objecter ms_handle_connect 0x7f06c8038410 2024-09-18T11:44:25.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7efd9effd700 20 client.4310.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f654dffb700 20 client.4316.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f655dccb700 10 monclient: get_auth_request con 0x7f6544040d80 auth_method 0 2024-09-18T11:44:25.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04540020e0 con 0x7f045c09f270 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0454005000 con 0x7f045c09f270 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f0458ff9700 10 monclient: dump: 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:25.789 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:25.790 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:25.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.694+0000 7f045affd700 4 set_mon_vals no callback set 2024-09-18T11:44:25.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0454005160 con 0x7f045c09f270 2024-09-18T11:44:25.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7efdb659c700 10 monclient: get_auth_request con 0x7efda4038740 auth_method 0 2024-09-18T11:44:25.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f0454043930 con 0x7f045c09f270 2024-09-18T11:44:25.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f0458ff9700 10 client.4299.objecter ms_dispatch 0x7f045c000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:25.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f0458ff9700 3 client.4299.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:25.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f0458ff9700 3 client.4299.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:25.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f0458ff9700 20 client.4299.objecter dump_active .. 0 homeless 2024-09-18T11:44:25.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f0460fd2700 10 monclient: get_auth_request con 0x7f0448038350 auth_method 0 2024-09-18T11:44:25.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f654dffb700 10 client.4316.objecter ms_handle_connect 0x7f6544040d80 2024-09-18T11:44:25.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.698+0000 7f0458ff9700 10 client.4299.objecter ms_handle_connect 0x7f0448038350 2024-09-18T11:44:25.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f65437fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f65437fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f65437fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f0463236700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f65437fe700 1 -- 172.21.15.112:0/2603400776 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65380050b0 con 0x7f65580a1230 2024-09-18T11:44:25.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f0463236700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f0463236700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043c0050b0 con 0x7f045c09f270 2024-09-18T11:44:25.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f06e6263700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f06e6263700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f06e6263700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.702+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c00050b0 con 0x7f06e006a8b0 2024-09-18T11:44:25.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7efdbd19d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7efdbd19d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7efdbd19d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd980050b0 con 0x7efdb806e620 2024-09-18T11:44:25.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f6550015190 con 0x7f65580a1230 2024-09-18T11:44:25.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7f654dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7f654dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:25.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f0454014c30 con 0x7f045c09f270 2024-09-18T11:44:25.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7f0458ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.706+0000 7f0458ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:25.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.710+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f06d8002fa0 con 0x7f06e006a8b0 2024-09-18T11:44:25.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.710+0000 7f06c7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.710+0000 7f06c7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:25.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.722+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7efdac0148e0 con 0x7efdb806e620 2024-09-18T11:44:25.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.722+0000 7efd9effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:25.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.722+0000 7efd9effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:25.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.722+0000 7efd9effd700 10 client.4310.objecter ms_handle_connect 0x7efda4038740 2024-09-18T11:44:25.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.842+0000 7ff36d9b9700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T11:44:25.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.842+0000 7ff36d9b9700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T11:44:25.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.842+0000 7ff36d9b9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.842+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff348005360 con 0x7ff368138ca0 2024-09-18T11:44:25.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.842+0000 7ff34effd700 1 -- 172.21.15.112:0/98841267 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 956180462) 0x7ff35c0145c0 con 0x7ff368138ca0 2024-09-18T11:44:25.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.842+0000 7ff34effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T11:44:25.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.842+0000 7ff34effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:25.848 INFO:teuthology.orchestra.run.smithi112.stdout:51539607555 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 10 librados: watch_flush enter 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 10 librados: watch_flush exit 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 20 client.4295.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 10 client.4295.objecter successfully canceled tick 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 >> v1:172.21.15.129:6812/20806 conn(0x7ff3540382a0 legacy=0x7ff35403a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 10 monclient: shutdown 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 >> v1:172.21.15.112:6790/0 conn(0x7ff368138ca0 legacy=0x7ff36814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 shutdown_connections 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 >> 172.21.15.112:0/98841267 conn(0x7ff368068110 msgr2=0x7ff36806d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 shutdown_connections 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 1 -- 172.21.15.112:0/98841267 wait complete. 2024-09-18T11:44:25.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.846+0000 7ff36d9b9700 1 librados: shutdown 2024-09-18T11:44:25.860 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.4 2024-09-18T11:44:25.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.870+0000 7f65437fe700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T11:44:25.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.870+0000 7f65437fe700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T11:44:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.870+0000 7f65437fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.870+0000 7f65437fe700 1 -- 172.21.15.112:0/2603400776 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6538005360 con 0x7f65580a1230 2024-09-18T11:44:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.870+0000 7f654dffb700 1 -- 172.21.15.112:0/2603400776 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 956180462) 0x7f6550014b90 con 0x7f65580a1230 2024-09-18T11:44:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.870+0000 7f654dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T11:44:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.870+0000 7f654dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:25.874 INFO:teuthology.orchestra.run.smithi112.stdout:51539607555 2024-09-18T11:44:25.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 10 librados: watch_flush enter 2024-09-18T11:44:25.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 10 librados: watch_flush exit 2024-09-18T11:44:25.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 20 client.4316.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 10 client.4316.objecter successfully canceled tick 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 1 -- 172.21.15.112:0/2603400776 >> v1:172.21.15.129:6812/20806 conn(0x7f6544040d80 legacy=0x7f6544043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 10 monclient: shutdown 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 1 -- 172.21.15.112:0/2603400776 >> v1:172.21.15.112:6790/0 conn(0x7f65580a1230 legacy=0x7f6558142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 1 -- 172.21.15.112:0/2603400776 shutdown_connections 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 1 -- 172.21.15.112:0/2603400776 >> 172.21.15.112:0/2603400776 conn(0x7f65580680d0 msgr2=0x7f655806d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 1 -- 172.21.15.112:0/2603400776 shutdown_connections 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 1 -- 172.21.15.112:0/2603400776 wait complete. 2024-09-18T11:44:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.874+0000 7f65437fe700 1 librados: shutdown 2024-09-18T11:44:25.886 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.2 2024-09-18T11:44:25.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.970+0000 7fbd99159700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T11:44:25.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.970+0000 7fbd99159700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T11:44:25.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.970+0000 7fbd99159700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.970+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbd78005360 con 0x7fbd94135220 2024-09-18T11:44:25.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.970+0000 7fbd8bfff700 1 -- 172.21.15.112:0/4038887935 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 956180462) 0x7fbd8c014a10 con 0x7fbd94135220 2024-09-18T11:44:25.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.970+0000 7fbd8bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T11:44:25.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.970+0000 7fbd8bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:25.976 INFO:teuthology.orchestra.run.smithi112.stdout:51539607555 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 10 librados: watch_flush enter 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 10 librados: watch_flush exit 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 20 client.4278.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 10 client.4278.objecter successfully canceled tick 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 >> v1:172.21.15.129:6812/20806 conn(0x7fbd740382c0 legacy=0x7fbd7403a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 10 monclient: shutdown 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 >> v1:172.21.15.112:6789/0 conn(0x7fbd94135220 legacy=0x7fbd941476d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 shutdown_connections 2024-09-18T11:44:25.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 >> 172.21.15.112:0/4038887935 conn(0x7fbd94067da0 msgr2=0x7fbd9406cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 shutdown_connections 2024-09-18T11:44:25.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 1 -- 172.21.15.112:0/4038887935 wait complete. 2024-09-18T11:44:25.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.974+0000 7fbd99159700 1 librados: shutdown 2024-09-18T11:44:25.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.978+0000 7efdbd19d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T11:44:25.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.978+0000 7efdbd19d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T11:44:25.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.978+0000 7efdbd19d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.978+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efd98005360 con 0x7efdb806e620 2024-09-18T11:44:25.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.978+0000 7efd9effd700 1 -- 172.21.15.112:0/4243722891 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 847495621) 0x7efdac0142e0 con 0x7efdb806e620 2024-09-18T11:44:25.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.978+0000 7efd9effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T11:44:25.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.978+0000 7efd9effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:25.983 INFO:teuthology.orchestra.run.smithi112.stdout:47244640259 2024-09-18T11:44:25.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 10 librados: watch_flush enter 2024-09-18T11:44:25.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 10 librados: watch_flush exit 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 20 client.4310.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 10 client.4310.objecter successfully canceled tick 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 >> v1:172.21.15.129:6812/20806 conn(0x7efda4038740 legacy=0x7efda403abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 10 monclient: shutdown 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 >> v1:172.21.15.112:6790/0 conn(0x7efdb806e620 legacy=0x7efdb8097d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 shutdown_connections 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 >> 172.21.15.112:0/4243722891 conn(0x7efdb806a230 msgr2=0x7efdb806d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 shutdown_connections 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 1 -- 172.21.15.112:0/4243722891 wait complete. 2024-09-18T11:44:25.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7efdbd19d700 1 librados: shutdown 2024-09-18T11:44:25.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7f06e6263700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T11:44:25.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7f06e6263700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T11:44:25.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7f06e6263700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:25.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.982+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f06c0005360 con 0x7f06e006a8b0 2024-09-18T11:44:25.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06c7fff700 1 -- 172.21.15.112:0/3872958422 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 555833778) 0x7f06d80149f0 con 0x7f06e006a8b0 2024-09-18T11:44:25.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06c7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T11:44:25.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06c7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:25.989 INFO:teuthology.orchestra.run.smithi112.stdout:47244640258 2024-09-18T11:44:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 10 librados: watch_flush enter 2024-09-18T11:44:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 10 librados: watch_flush exit 2024-09-18T11:44:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 20 client.4313.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 10 client.4313.objecter successfully canceled tick 2024-09-18T11:44:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 >> v1:172.21.15.129:6812/20806 conn(0x7f06c8038410 legacy=0x7f06c803a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 10 monclient: shutdown 2024-09-18T11:44:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 >> v1:172.21.15.112:6790/0 conn(0x7f06e006a8b0 legacy=0x7f06e009dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 shutdown_connections 2024-09-18T11:44:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 >> 172.21.15.112:0/3872958422 conn(0x7f06e00a5b70 msgr2=0x7f06e013e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 shutdown_connections 2024-09-18T11:44:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 1 -- 172.21.15.112:0/3872958422 wait complete. 2024-09-18T11:44:25.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.986+0000 7f06e6263700 1 librados: shutdown 2024-09-18T11:44:25.993 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.3 2024-09-18T11:44:25.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.990+0000 7f0463236700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T11:44:25.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.990+0000 7f0463236700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T11:44:25.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.990+0000 7f0463236700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:25.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.990+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f043c005360 con 0x7f045c09f270 2024-09-18T11:44:25.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.990+0000 7f0458ff9700 1 -- 172.21.15.112:0/3440591962 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 956180462) 0x7f0454014630 con 0x7f045c09f270 2024-09-18T11:44:25.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.990+0000 7f0458ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T11:44:25.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.990+0000 7f0458ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:25.996 INFO:teuthology.orchestra.run.smithi112.stdout:51539607555 2024-09-18T11:44:25.997 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.1 2024-09-18T11:44:25.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 10 librados: watch_flush enter 2024-09-18T11:44:25.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 10 librados: watch_flush exit 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 20 client.4299.objecter shutdown clearing up homeless session... 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 10 client.4299.objecter successfully canceled tick 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 >> v1:172.21.15.129:6812/20806 conn(0x7f0448038350 legacy=0x7f044803a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 10 monclient: shutdown 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 >> v1:172.21.15.112:6789/0 conn(0x7f045c09f270 legacy=0x7f045c14de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 shutdown_connections 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 >> 172.21.15.112:0/3440591962 conn(0x7f045c0a5b70 msgr2=0x7f045c139570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 shutdown_connections 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 1 -- 172.21.15.112:0/3440591962 wait complete. 2024-09-18T11:44:25.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:25.994+0000 7f0463236700 1 librados: shutdown 2024-09-18T11:44:26.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2024-09-18T11:44:26.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.5 2024-09-18T11:44:26.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T11:44:26.887 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:44:26.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient: get_monmap_and_config 2024-09-18T11:44:26.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient: build_initial_monmap 2024-09-18T11:44:26.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient: monmap: 2024-09-18T11:44:26.981 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:26.981 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:26.981 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:26.981531+0000 2024-09-18T11:44:26.982 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:26.981531+0000 2024-09-18T11:44:26.982 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:26.982 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:26.982 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:26.982 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:26.982 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:26.982 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:26.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 1 Processor -- start 2024-09-18T11:44:26.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 1 -- start start 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient: init 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient(hunting): picked mon.noname-b con 0x7f4c50067680 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient(hunting): picked mon.noname-a con 0x7f4c5006f9e0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient(hunting): picked mon.noname-c con 0x7f4c50136650 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50067a50 con 0x7f4c5006f9e0 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5009d640 con 0x7f4c50136650 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5009d870 con 0x7f4c50067680 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c551ad700 10 monclient(hunting): authenticate will time out at 838.823367s 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c4ffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f4c50067680 0x7f4c5009c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41588/0 (socket says 172.21.15.112:41588) 2024-09-18T11:44:26.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c4ffff700 1 -- 172.21.15.112:0/1440496919 learned_addr learned my addr 172.21.15.112:0/1440496919 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:26.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.978+0000 7f4c47fff700 1 --1- 172.21.15.112:0/1440496919 >> v1:172.21.15.112:6789/0 conn(0x7f4c5006f9e0 0x7f4c5009cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35266/0 (socket says 172.21.15.112:35266) 2024-09-18T11:44:26.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c47fff700 1 -- 172.21.15.112:0/1440496919 >> v1:172.21.15.112:6789/0 conn(0x7f4c5006f9e0 legacy=0x7f4c5009cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:26.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4ffff700 1 -- 172.21.15.112:0/1440496919 >> v1:172.21.15.112:6790/0 conn(0x7f4c50067680 legacy=0x7f4c5009c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566469918 0 0) 0x7f4c5009d870 con 0x7f4c50067680 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c549ac700 1 -- 172.21.15.112:0/1440496919 >> v1:172.21.15.129:6789/0 conn(0x7f4c50136650 legacy=0x7f4c50138a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient(hunting): my global_id is 4322 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c003570 con 0x7f4c50067680 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376763738 0 0) 0x7f4c50067a50 con 0x7f4c5006f9e0 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient(hunting): my global_id is 4305 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c5009d870 con 0x7f4c5006f9e0 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1159417097 0 0) 0x7f4c5009d870 con 0x7f4c5006f9e0 2024-09-18T11:44:26.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 >> v1:172.21.15.112:6790/0 conn(0x7f4c50067680 legacy=0x7f4c5009c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 >> v1:172.21.15.129:6789/0 conn(0x7f4c50136650 legacy=0x7f4c50138a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 monclient: found mon.noname-a 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5009c280 con 0x7f4c5006f9e0 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4c380019e0 con 0x7f4c5006f9e0 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:26.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: dump: 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 5 monclient: authenticate success, global_id 4305 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c38001bc0 con 0x7f4c5006f9e0 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:26.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 1 -- 172.21.15.112:0/1440496919 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4c380049d0 con 0x7f4c5006f9e0 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 4 set_mon_vals no callback set 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c4f7fe700 10 monclient: dump: 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:26.990 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:26.991 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:26.991 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:26.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 10 monclient: shutdown 2024-09-18T11:44:26.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:26.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 1 -- 172.21.15.112:0/1440496919 >> v1:172.21.15.112:6789/0 conn(0x7f4c5006f9e0 legacy=0x7f4c5009cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:26.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 1 -- 172.21.15.112:0/1440496919 shutdown_connections 2024-09-18T11:44:26.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 1 -- 172.21.15.112:0/1440496919 >> 172.21.15.112:0/1440496919 conn(0x7f4c500a1e30 msgr2=0x7f4c500a2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:26.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 1 -- 172.21.15.112:0/1440496919 shutdown_connections 2024-09-18T11:44:26.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 1 -- 172.21.15.112:0/1440496919 wait complete. 2024-09-18T11:44:26.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7ff3dd252700 10 monclient: get_monmap_and_config 2024-09-18T11:44:26.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7ff3dd252700 10 monclient: build_initial_monmap 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 10 monclient: build_initial_monmap 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7f4c551ad700 10 monclient: monmap: 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:26.993 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.982+0000 7ff3dd252700 10 monclient: monmap: 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:26.988263+0000 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:26.988263+0000 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:26.995 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:26.995 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:26.995 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:26.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 librados: starting msgr at 2024-09-18T11:44:26.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 librados: starting objecter 2024-09-18T11:44:26.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 Processor -- start 2024-09-18T11:44:26.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 -- start start 2024-09-18T11:44:26.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 librados: setting wanted keys 2024-09-18T11:44:26.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 librados: calling monclient init 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 10 monclient: init 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 10 monclient(hunting): picked mon.noname-a con 0x7f4c50067680 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 10 monclient(hunting): picked mon.noname-c con 0x7f4c5006f9e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 10 monclient(hunting): picked mon.noname-b con 0x7f4c50136650 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50132210 con 0x7f4c50067680 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50132780 con 0x7f4c50136650 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5009e1e0 con 0x7f4c5006f9e0 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c551ad700 10 monclient(hunting): authenticate will time out at 838.828871s 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4ffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f4c50067680 0x7f4c5006d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35282/0 (socket says 172.21.15.112:35282) 2024-09-18T11:44:26.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4ffff700 1 -- 172.21.15.112:0/98557609 learned_addr learned my addr 172.21.15.112:0/98557609 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:26.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c47fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f4c5006f9e0 0x7f4c5006d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41592/0 (socket says 172.21.15.112:41592) 2024-09-18T11:44:26.998 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T11:44:26.998 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:44:26.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 1 Processor -- start 2024-09-18T11:44:26.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 1 -- start start 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 10 monclient: init 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 10 monclient(hunting): picked mon.noname-c con 0x7ff3d8067680 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 10 monclient(hunting): picked mon.noname-b con 0x7ff3d806f800 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 10 monclient(hunting): picked mon.noname-a con 0x7ff3d81357c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d8098dc0 con 0x7ff3d81357c0 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d8067a50 con 0x7ff3d8067680 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d8098a70 con 0x7ff3d806f800 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3dd252700 10 monclient(hunting): authenticate will time out at 838.829715s 2024-09-18T11:44:27.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3d759e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff3d81357c0 0x7ff3d8137ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35284/0 (socket says 172.21.15.112:35284) 2024-09-18T11:44:27.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3d759e700 1 -- 172.21.15.112:0/1556641713 learned_addr learned my addr 172.21.15.112:0/1556641713 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4ffff700 1 -- 172.21.15.112:0/98557609 >> v1:172.21.15.112:6789/0 conn(0x7f4c50067680 legacy=0x7f4c5006d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 client.?.objecter ms_handle_connect 0x7f4c50067680 2024-09-18T11:44:27.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 client.?.objecter ms_handle_connect 0x7f4c5006f9e0 2024-09-18T11:44:27.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c47fff700 1 -- 172.21.15.112:0/98557609 >> v1:172.21.15.112:6790/0 conn(0x7f4c5006f9e0 legacy=0x7f4c5006d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174698418 0 0) 0x7f4c50132210 con 0x7f4c50067680 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): my global_id is 4308 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c30004f10 con 0x7f4c50067680 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574473618 0 0) 0x7f4c5009e1e0 con 0x7f4c5006f9e0 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): my global_id is 4325 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50132210 con 0x7f4c5006f9e0 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 client.?.objecter ms_handle_connect 0x7f4c50136650 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c549ac700 1 -- 172.21.15.112:0/98557609 >> v1:172.21.15.129:6789/0 conn(0x7f4c50136650 legacy=0x7f4c5006e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3490494425 0 0) 0x7f4c30004f10 con 0x7f4c50067680 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c5009e1e0 con 0x7f4c50067680 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4c40004830 con 0x7f4c50067680 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): dump: 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.004 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547641978 0 0) 0x7f4c50132780 con 0x7f4c50136650 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 10 monclient(hunting): my global_id is 4315 2024-09-18T11:44:27.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c30004f10 con 0x7f4c50136650 2024-09-18T11:44:27.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:44:27.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7ff3d759e700 1 -- 172.21.15.112:0/1556641713 >> v1:172.21.15.112:6789/0 conn(0x7ff3d81357c0 legacy=0x7ff3d8137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 643707145 0 0) 0x7f4c5009e1e0 con 0x7f4c50067680 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.986+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 >> v1:172.21.15.112:6790/0 conn(0x7f4c5006f9e0 legacy=0x7f4c5006d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 >> v1:172.21.15.129:6789/0 conn(0x7f4c50136650 legacy=0x7f4c5006e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 1 monclient: found mon.a 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5011e1b0 con 0x7f4c50067680 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 5 monclient: authenticate success, global_id 4308 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 10 monclient: _renew_subs 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5006ed30 con 0x7f4c50067680 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 10 monclient: _renew_subs 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5011d460 con 0x7f4c50067680 2024-09-18T11:44:27.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 1 librados: init done 2024-09-18T11:44:27.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d659c700 1 -- 172.21.15.112:0/1556641713 >> v1:172.21.15.112:6790/0 conn(0x7ff3d806f800 legacy=0x7ff3d8097170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924513944 0 0) 0x7ff3d8098dc0 con 0x7ff3d81357c0 2024-09-18T11:44:27.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient(hunting): my global_id is 4311 2024-09-18T11:44:27.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c0003570 con 0x7ff3d81357c0 2024-09-18T11:44:27.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103852730 0 0) 0x7ff3d8098a70 con 0x7ff3d806f800 2024-09-18T11:44:27.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient(hunting): my global_id is 4328 2024-09-18T11:44:27.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8098dc0 con 0x7ff3d806f800 2024-09-18T11:44:27.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c40004cb0 con 0x7f4c50067680 2024-09-18T11:44:27.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4c40004fc0 con 0x7f4c50067680 2024-09-18T11:44:27.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4effd700 4 set_mon_vals no callback set 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 10 monclient: dump: 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.011 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3425265567 0 0) 0x7ff3d8098dc0 con 0x7ff3d806f800 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 >> v1:172.21.15.129:6789/0 conn(0x7ff3d8067680 legacy=0x7ff3d8096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 >> v1:172.21.15.112:6789/0 conn(0x7ff3d81357c0 legacy=0x7ff3d8137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 monclient: found mon.noname-b 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8097880 con 0x7ff3d806f800 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3cc001a30 con 0x7ff3d806f800 2024-09-18T11:44:27.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 5 monclient: authenticate success, global_id 4328 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: dump: 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.014 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3cc001c10 con 0x7ff3d806f800 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 1 -- 172.21.15.112:0/1556641713 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3cc0020a0 con 0x7ff3d806f800 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 4 set_mon_vals no callback set 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3d5d9b700 10 monclient: dump: 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.015 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.016 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.016 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.016 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.016 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.016 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.016 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T11:44:27.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f4c4000f840 con 0x7f4c50067680 2024-09-18T11:44:27.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c340050b0 con 0x7f4c50067680 2024-09-18T11:44:27.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f4c400438a0 con 0x7f4c50067680 2024-09-18T11:44:27.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 10 client.4308.objecter ms_dispatch 0x7f4c50000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:27.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 3 client.4308.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:27.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 3 client.4308.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:27.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c4cff9700 20 client.4308.objecter dump_active .. 0 homeless 2024-09-18T11:44:27.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient: shutdown 2024-09-18T11:44:27.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- 172.21.15.112:0/1556641713 >> v1:172.21.15.112:6790/0 conn(0x7ff3d806f800 legacy=0x7ff3d8097170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- 172.21.15.112:0/1556641713 shutdown_connections 2024-09-18T11:44:27.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- 172.21.15.112:0/1556641713 >> 172.21.15.112:0/1556641713 conn(0x7ff3d80a1fb0 msgr2=0x7ff3d80a23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7f4c47fff700 10 monclient: get_auth_request con 0x7f4c300383f0 auth_method 0 2024-09-18T11:44:27.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- 172.21.15.112:0/1556641713 shutdown_connections 2024-09-18T11:44:27.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- 172.21.15.112:0/1556641713 wait complete. 2024-09-18T11:44:27.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient: monmap: 2024-09-18T11:44:27.021 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.021 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:27.021 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 librados: starting msgr at 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 librados: starting objecter 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 Processor -- start 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- start start 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 librados: setting wanted keys 2024-09-18T11:44:27.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 librados: calling monclient init 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient: init 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient(hunting): picked mon.noname-b con 0x7ff3d8067680 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient(hunting): picked mon.noname-c con 0x7ff3d806f800 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 10 monclient(hunting): picked mon.noname-a con 0x7ff3d81357c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d811dd60 con 0x7ff3d81357c0 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d811d380 con 0x7ff3d8067680 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.990+0000 7ff3dd252700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d8097a70 con 0x7ff3d806f800 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff3d806f800 0x7ff3d8145c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41610/0 (socket says 172.21.15.112:41610) 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d659c700 1 -- 172.21.15.112:0/1679192423 learned_addr learned my addr 172.21.15.112:0/1679192423 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 10 monclient(hunting): authenticate will time out at 838.836135s 2024-09-18T11:44:27.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d6d9d700 1 -- 172.21.15.112:0/1679192423 >> v1:172.21.15.129:6789/0 conn(0x7ff3d8067680 legacy=0x7ff3d8130160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T11:44:27.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d6d9d700 1 -- 172.21.15.112:0/1679192423 >> v1:172.21.15.129:6789/0 conn(0x7ff3d8067680 legacy=0x7ff3d8130160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:44:27.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d6d9d700 1 --1- 172.21.15.112:0/1679192423 >> v1:172.21.15.129:6789/0 conn(0x7ff3d8067680 0x7ff3d8130160 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-18T11:44:27.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d6d9d700 1 --1- 172.21.15.112:0/1679192423 >> v1:172.21.15.129:6789/0 conn(0x7ff3d8067680 0x7ff3d8130160 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-18T11:44:27.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.129:6789/0 2024-09-18T11:44:27.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7f4c4cff9700 10 client.4308.objecter ms_handle_connect 0x7f4c300383f0 2024-09-18T11:44:27.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d659c700 1 -- 172.21.15.112:0/1679192423 >> v1:172.21.15.112:6790/0 conn(0x7ff3d806f800 legacy=0x7ff3d8145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 client.?.objecter ms_handle_connect 0x7ff3d806f800 2024-09-18T11:44:27.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 client.?.objecter ms_handle_connect 0x7ff3d81357c0 2024-09-18T11:44:27.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d759e700 1 -- 172.21.15.112:0/1679192423 >> v1:172.21.15.112:6789/0 conn(0x7ff3d81357c0 legacy=0x7ff3d81493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f4c40014ba0 con 0x7f4c50067680 2024-09-18T11:44:27.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7f4c4cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7f4c4cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:27.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129000322 0 0) 0x7ff3d8097a70 con 0x7ff3d806f800 2024-09-18T11:44:27.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): my global_id is 4331 2024-09-18T11:44:27.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c4004f10 con 0x7ff3d806f800 2024-09-18T11:44:27.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430591293 0 0) 0x7ff3d811dd60 con 0x7ff3d81357c0 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): my global_id is 4314 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8097a70 con 0x7ff3d81357c0 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3608422845 0 0) 0x7ff3c4004f10 con 0x7ff3d806f800 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3d811dd60 con 0x7ff3d806f800 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3cc002160 con 0x7ff3d806f800 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): dump: 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.030 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3441520852 0 0) 0x7ff3d8097a70 con 0x7ff3d81357c0 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3c4004f10 con 0x7ff3d81357c0 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3d0001f90 con 0x7ff3d81357c0 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient(hunting): dump: 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1130189221 0 0) 0x7ff3d811dd60 con 0x7ff3d806f800 2024-09-18T11:44:27.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 >> v1:172.21.15.129:6789/0 conn(0x7ff3d8067680 legacy=0x7ff3d8130160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 >> v1:172.21.15.112:6789/0 conn(0x7ff3d81357c0 legacy=0x7ff3d81493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 monclient: found mon.c 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d811d0a0 con 0x7ff3d806f800 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 5 monclient: authenticate success, global_id 4331 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 10 monclient: _renew_subs 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d812f550 con 0x7ff3d806f800 2024-09-18T11:44:27.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 10 monclient: _renew_subs 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d806e970 con 0x7ff3d806f800 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3dd252700 1 librados: init done 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3cc002750 con 0x7ff3d806f800 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3cc002a60 con 0x7ff3d806f800 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3d559a700 4 set_mon_vals no callback set 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.994+0000 7ff3beffd700 10 monclient: dump: 2024-09-18T11:44:27.034 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ff3cc00ff30 con 0x7ff3d806f800 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3dd252700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3dd252700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3dd252700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b80050b0 con 0x7ff3d806f800 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7ff3cc043a20 con 0x7ff3d806f800 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3beffd700 10 client.4331.objecter ms_dispatch 0x7ff3d8000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3beffd700 3 client.4331.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3beffd700 3 client.4331.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3beffd700 20 client.4331.objecter dump_active .. 0 homeless 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3d6d9d700 10 monclient: get_auth_request con 0x7ff3c4038300 auth_method 0 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:26.998+0000 7ff3beffd700 10 client.4331.objecter ms_handle_connect 0x7ff3c4038300 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.006+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7ff3cc014de0 con 0x7ff3d806f800 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.006+0000 7ff3beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.006+0000 7ff3beffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:27.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient: get_monmap_and_config 2024-09-18T11:44:27.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient: monmap: 2024-09-18T11:44:27.122 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.122 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.122 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:27.121430+0000 2024-09-18T11:44:27.122 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:27.121430+0000 2024-09-18T11:44:27.122 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.123 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.123 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.123 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:27.123 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:27.123 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 1 Processor -- start 2024-09-18T11:44:27.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 1 -- start start 2024-09-18T11:44:27.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient: init 2024-09-18T11:44:27.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:27.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient(hunting): picked mon.noname-c con 0x7f3cdc09ae70 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient(hunting): picked mon.noname-b con 0x7f3cdc132860 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient(hunting): picked mon.noname-a con 0x7f3cdc132c60 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc06e750 con 0x7f3cdc132c60 2024-09-18T11:44:27.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc09b240 con 0x7f3cdc09ae70 2024-09-18T11:44:27.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc09b660 con 0x7f3cdc132860 2024-09-18T11:44:27.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3ce1eb6700 10 monclient(hunting): authenticate will time out at 838.962752s 2024-09-18T11:44:27.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3cdaffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3cdc132860 0x7f3cdc137480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41616/0 (socket says 172.21.15.112:41616) 2024-09-18T11:44:27.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3cdaffd700 1 -- 172.21.15.112:0/3619165815 learned_addr learned my addr 172.21.15.112:0/3619165815 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.118+0000 7f3cdbfff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3cdc132c60 0x7f3cdc13abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35296/0 (socket says 172.21.15.112:35296) 2024-09-18T11:44:27.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cdbfff700 1 -- 172.21.15.112:0/3619165815 >> v1:172.21.15.112:6789/0 conn(0x7f3cdc132c60 legacy=0x7f3cdc13abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cdaffd700 1 -- 172.21.15.112:0/3619165815 >> v1:172.21.15.112:6790/0 conn(0x7f3cdc132860 legacy=0x7f3cdc137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cdb7fe700 1 -- 172.21.15.112:0/3619165815 >> v1:172.21.15.129:6789/0 conn(0x7f3cdc09ae70 legacy=0x7f3cdc132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506822891 0 0) 0x7f3cdc09b660 con 0x7f3cdc132860 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): my global_id is 4334 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cc4003570 con 0x7f3cdc132860 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942063419 0 0) 0x7f3cdc06e750 con 0x7f3cdc132c60 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): my global_id is 4317 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc09b660 con 0x7f3cdc132c60 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330370824 0 0) 0x7f3cdc09b240 con 0x7f3cdc09ae70 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient(hunting): my global_id is 4318 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc06e750 con 0x7f3cdc09ae70 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3936511021 0 0) 0x7f3cc4003570 con 0x7f3cdc132860 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 >> v1:172.21.15.129:6789/0 conn(0x7f3cdc09ae70 legacy=0x7f3cdc132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 >> v1:172.21.15.112:6789/0 conn(0x7f3cdc132c60 legacy=0x7f3cdc13abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 monclient: found mon.noname-b 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc09b8a0 con 0x7f3cdc132860 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3ccc001940 con 0x7f3cdc132860 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: dump: 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.130 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 5 monclient: authenticate success, global_id 4334 2024-09-18T11:44:27.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ccc001bd0 con 0x7f3cdc132860 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 1 -- 172.21.15.112:0/3619165815 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3ccc004a90 con 0x7f3cdc132860 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 4 set_mon_vals no callback set 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3cda7fc700 10 monclient: dump: 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.132 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 10 monclient: shutdown 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/3619165815 >> v1:172.21.15.112:6790/0 conn(0x7f3cdc132860 legacy=0x7f3cdc137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/3619165815 shutdown_connections 2024-09-18T11:44:27.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.122+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/3619165815 >> 172.21.15.112:0/3619165815 conn(0x7f3cdc0680d0 msgr2=0x7f3cdc0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.126+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/3619165815 shutdown_connections 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.126+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/3619165815 wait complete. 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient: monmap: 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:27.135 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 librados: starting msgr at 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 librados: starting objecter 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 Processor -- start 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 -- start start 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 librados: setting wanted keys 2024-09-18T11:44:27.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 librados: calling monclient init 2024-09-18T11:44:27.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient: init 2024-09-18T11:44:27.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:27.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient(hunting): picked mon.noname-b con 0x7f3cdc09ae70 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient: get_monmap_and_config 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient: monmap: 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:27.133139+0000 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:27.133139+0000 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:27.138 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:27.139 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient(hunting): picked mon.noname-a con 0x7f3cdc132860 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient(hunting): picked mon.noname-c con 0x7f3cdc132c60 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc1323a0 con 0x7f3cdc132860 2024-09-18T11:44:27.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc141bd0 con 0x7f3cdc09ae70 2024-09-18T11:44:27.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc142330 con 0x7f3cdc132c60 2024-09-18T11:44:27.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3ce1eb6700 10 monclient(hunting): authenticate will time out at 838.973752s 2024-09-18T11:44:27.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3cdbfff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3cdc132c60 0x7f3cdc149850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41618/0 (socket says 172.21.15.112:41618) 2024-09-18T11:44:27.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f3cdbfff700 1 -- 172.21.15.112:0/829455225 learned_addr learned my addr 172.21.15.112:0/829455225 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 1 Processor -- start 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 1 -- start start 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient: init 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f53b80972f0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f53b812fdd0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f53b81308e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b8097ea0 con 0x7f53b80972f0 2024-09-18T11:44:27.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b8096ac0 con 0x7f53b812fdd0 2024-09-18T11:44:27.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b8098180 con 0x7f53b81308e0 2024-09-18T11:44:27.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.130+0000 7f53b7fff700 10 monclient(hunting): authenticate will time out at 838.974726s 2024-09-18T11:44:27.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f53b6ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f53b80972f0 0x7f53b8098c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35306/0 (socket says 172.21.15.112:35306) 2024-09-18T11:44:27.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f53b6ffd700 1 -- 172.21.15.112:0/3026535388 learned_addr learned my addr 172.21.15.112:0/3026535388 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cdbfff700 1 -- 172.21.15.112:0/829455225 >> v1:172.21.15.112:6790/0 conn(0x7f3cdc132c60 legacy=0x7f3cdc149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 client.?.objecter ms_handle_connect 0x7f3cdc132c60 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cdaffd700 1 -- 172.21.15.112:0/829455225 >> v1:172.21.15.112:6789/0 conn(0x7f3cdc132860 legacy=0x7f3cdc131490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 client.?.objecter ms_handle_connect 0x7f3cdc132860 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cdb7fe700 1 -- 172.21.15.112:0/829455225 >> v1:172.21.15.129:6789/0 conn(0x7f3cdc09ae70 legacy=0x7f3cdc1414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 client.?.objecter ms_handle_connect 0x7f3cdc09ae70 2024-09-18T11:44:27.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690439719 0 0) 0x7f3cdc142330 con 0x7f3cdc132c60 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): my global_id is 4337 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cc8004f10 con 0x7f3cdc132c60 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263392133 0 0) 0x7f3cdc1323a0 con 0x7f3cdc132860 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): my global_id is 4320 2024-09-18T11:44:27.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc142330 con 0x7f3cdc132860 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565325387 0 0) 0x7f3cdc141bd0 con 0x7f3cdc09ae70 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): my global_id is 4321 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc1323a0 con 0x7f3cdc09ae70 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657801253 0 0) 0x7f3cc8004f10 con 0x7f3cdc132c60 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cdc141bd0 con 0x7f3cdc132c60 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3cd0004830 con 0x7f3cdc132c60 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): dump: 2024-09-18T11:44:27.146 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:27.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3818942146 0 0) 0x7f3cdc141bd0 con 0x7f3cdc132c60 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 >> v1:172.21.15.129:6789/0 conn(0x7f3cdc09ae70 legacy=0x7f3cdc1414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 >> v1:172.21.15.112:6789/0 conn(0x7f3cdc132860 legacy=0x7f3cdc131490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 monclient: found mon.c 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc11e5b0 con 0x7f3cdc132c60 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 5 monclient: authenticate success, global_id 4337 2024-09-18T11:44:27.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 10 monclient: _renew_subs 2024-09-18T11:44:27.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc131ba0 con 0x7f3cdc132c60 2024-09-18T11:44:27.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:27.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 10 monclient: _renew_subs 2024-09-18T11:44:27.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc11d7f0 con 0x7f3cdc132c60 2024-09-18T11:44:27.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 1 librados: init done 2024-09-18T11:44:27.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f53b77fe700 1 -- 172.21.15.112:0/3026535388 >> v1:172.21.15.112:6790/0 conn(0x7f53b81308e0 legacy=0x7f53b8138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cd0004c40 con 0x7f3cdc132c60 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3cd0004f50 con 0x7f3cdc132c60 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 monclient: dump: 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.150 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cd9ffb700 4 set_mon_vals no callback set 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f3cd000f7d0 con 0x7f3cdc132c60 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f3cd00437c0 con 0x7f3cdc132c60 2024-09-18T11:44:27.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 10 client.4337.objecter ms_dispatch 0x7f3cdc000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:27.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 3 client.4337.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:27.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 3 client.4337.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:27.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f53b67fc700 1 -- 172.21.15.112:0/3026535388 >> v1:172.21.15.129:6789/0 conn(0x7f53b812fdd0 legacy=0x7f53b81301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f53b6ffd700 1 -- 172.21.15.112:0/3026535388 >> v1:172.21.15.112:6789/0 conn(0x7f53b80972f0 legacy=0x7f53b8098c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc0050b0 con 0x7f3cdc132c60 2024-09-18T11:44:27.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cc37fe700 20 client.4337.objecter dump_active .. 0 homeless 2024-09-18T11:44:27.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.134+0000 7f3cdb7fe700 10 monclient: get_auth_request con 0x7f3cc80383a0 auth_method 0 2024-09-18T11:44:27.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.138+0000 7f3cc37fe700 10 client.4337.objecter ms_handle_connect 0x7f3cc80383a0 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55144371 0 0) 0x7f53b8098180 con 0x7f53b81308e0 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): my global_id is 4340 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a0003570 con 0x7f53b81308e0 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091439586 0 0) 0x7f53b8096ac0 con 0x7f53b812fdd0 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): my global_id is 4324 2024-09-18T11:44:27.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b8098180 con 0x7f53b812fdd0 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318437969 0 0) 0x7f53b8097ea0 con 0x7f53b80972f0 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient(hunting): my global_id is 4323 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b8096ac0 con 0x7f53b80972f0 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2904257219 0 0) 0x7f53b8098180 con 0x7f53b812fdd0 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 >> v1:172.21.15.112:6790/0 conn(0x7f53b81308e0 legacy=0x7f53b8138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 >> v1:172.21.15.112:6789/0 conn(0x7f53b80972f0 legacy=0x7f53b8098c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 monclient: found mon.noname-c 2024-09-18T11:44:27.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:27.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b813ca80 con 0x7f53b812fdd0 2024-09-18T11:44:27.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f53ac001820 con 0x7f53b812fdd0 2024-09-18T11:44:27.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: dump: 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.158 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 5 monclient: authenticate success, global_id 4324 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53ac001ab0 con 0x7f53b812fdd0 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 1 -- 172.21.15.112:0/3026535388 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f53ac001f00 con 0x7f53b812fdd0 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 4 set_mon_vals no callback set 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b5ffb700 10 monclient: dump: 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.159 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.160 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.160 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.160 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.160 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.160 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.160 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.160 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.160 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f3cd00146a0 con 0x7f3cdc132c60 2024-09-18T11:44:27.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f3cc37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f3cc37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:27.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 10 monclient: shutdown 2024-09-18T11:44:27.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 -- 172.21.15.112:0/3026535388 >> v1:172.21.15.129:6789/0 conn(0x7f53b812fdd0 legacy=0x7f53b81301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 -- 172.21.15.112:0/3026535388 shutdown_connections 2024-09-18T11:44:27.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 -- 172.21.15.112:0/3026535388 >> 172.21.15.112:0/3026535388 conn(0x7f53b8067da0 msgr2=0x7f53b80a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 -- 172.21.15.112:0/3026535388 shutdown_connections 2024-09-18T11:44:27.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 -- 172.21.15.112:0/3026535388 wait complete. 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 10 monclient: monmap: 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:27.167 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 librados: starting msgr at 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 librados: starting objecter 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 Processor -- start 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 -- start start 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 librados: setting wanted keys 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 1 librados: calling monclient init 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.142+0000 7f53b7fff700 10 monclient: init 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:27.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f53b80972f0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f53b812fdd0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f53b81308e0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b806dee0 con 0x7f53b80972f0 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b8136a10 con 0x7f53b81308e0 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b8136cf0 con 0x7f53b812fdd0 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b7fff700 10 monclient(hunting): authenticate will time out at 838.988737s 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b6ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f53b80972f0 0x7f53b806d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35314/0 (socket says 172.21.15.112:35314) 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b67fc700 1 --1- 172.21.15.112:0/1444722537 >> v1:172.21.15.112:6790/0 conn(0x7f53b812fdd0 0x7f53b806d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41632/0 (socket says 172.21.15.112:41632) 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b6ffd700 1 -- 172.21.15.112:0/1444722537 learned_addr learned my addr 172.21.15.112:0/1444722537 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b6ffd700 1 -- 172.21.15.112:0/1444722537 >> v1:172.21.15.112:6789/0 conn(0x7f53b80972f0 legacy=0x7f53b806d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b67fc700 1 -- 172.21.15.112:0/1444722537 >> v1:172.21.15.112:6790/0 conn(0x7f53b812fdd0 legacy=0x7f53b806d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.146+0000 7f53b77fe700 1 -- 172.21.15.112:0/1444722537 >> v1:172.21.15.129:6789/0 conn(0x7f53b81308e0 legacy=0x7f53b8149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 client.?.objecter ms_handle_connect 0x7f53b80972f0 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 client.?.objecter ms_handle_connect 0x7f53b812fdd0 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 client.?.objecter ms_handle_connect 0x7f53b81308e0 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465216858 0 0) 0x7f53b806dee0 con 0x7f53b80972f0 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): my global_id is 4326 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a00031b0 con 0x7f53b80972f0 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156501958 0 0) 0x7f53b8136cf0 con 0x7f53b812fdd0 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): my global_id is 4343 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b806dee0 con 0x7f53b812fdd0 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814003261 0 0) 0x7f53b8136a10 con 0x7f53b81308e0 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): my global_id is 4327 2024-09-18T11:44:27.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b8136cf0 con 0x7f53b81308e0 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2953290768 0 0) 0x7f53a00031b0 con 0x7f53b80972f0 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53b8136a10 con 0x7f53b80972f0 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f53b0002100 con 0x7f53b80972f0 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): dump: 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.172 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3212420954 0 0) 0x7f53b806dee0 con 0x7f53b812fdd0 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53a00031b0 con 0x7f53b812fdd0 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f53ac001c20 con 0x7f53b812fdd0 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient(hunting): dump: 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2894898807 0 0) 0x7f53b8136a10 con 0x7f53b80972f0 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 >> v1:172.21.15.112:6790/0 conn(0x7f53b812fdd0 legacy=0x7f53b806d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 >> v1:172.21.15.129:6789/0 conn(0x7f53b81308e0 legacy=0x7f53b8149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 monclient: found mon.a 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b81372b0 con 0x7f53b80972f0 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 5 monclient: authenticate success, global_id 4326 2024-09-18T11:44:27.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 10 monclient: _renew_subs 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b8138dc0 con 0x7f53b80972f0 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 10 monclient: _renew_subs 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b81393b0 con 0x7f53b80972f0 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b7fff700 1 librados: init done 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53b0004e60 con 0x7f53b80972f0 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f53b0005170 con 0x7f53b80972f0 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b57fa700 4 set_mon_vals no callback set 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 monclient: dump: 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.177 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f53b000fbe0 con 0x7f53b80972f0 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f53b0043af0 con 0x7f53b80972f0 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 10 client.4326.objecter ms_dispatch 0x7f53b8000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 3 client.4326.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 3 client.4326.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f539effd700 20 client.4326.objecter dump_active .. 0 homeless 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.150+0000 7f53b67fc700 10 monclient: get_auth_request con 0x7f53a0038a20 auth_method 0 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.154+0000 7f539effd700 10 client.4326.objecter ms_handle_connect 0x7f53a0038a20 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.154+0000 7f53b7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.154+0000 7f53b7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.154+0000 7f53b7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.154+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53980050b0 con 0x7f53b80972f0 2024-09-18T11:44:27.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.154+0000 7f20858dd700 10 monclient: get_monmap_and_config 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.154+0000 7f20858dd700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.154+0000 7f20858dd700 10 monclient: monmap: 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:27.160171+0000 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:27.160171+0000 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:27.182 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 1 Processor -- start 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 1 -- start start 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 10 monclient: init 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 10 monclient(hunting): picked mon.noname-a con 0x7f2080067680 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 10 monclient(hunting): picked mon.noname-b con 0x7f20800987f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 10 monclient(hunting): picked mon.noname-c con 0x7f2080098bf0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2080132990 con 0x7f2080067680 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2080067a50 con 0x7f2080098bf0 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2080096a60 con 0x7f20800987f0 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f20858dd700 10 monclient(hunting): authenticate will time out at 839.001645s 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f207e7fc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f20800987f0 0x7f2080137400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41638/0 (socket says 172.21.15.112:41638) 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.158+0000 7f207e7fc700 1 -- 172.21.15.112:0/3085202964 learned_addr learned my addr 172.21.15.112:0/3085202964 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207e7fc700 1 -- 172.21.15.112:0/3085202964 >> v1:172.21.15.112:6790/0 conn(0x7f20800987f0 legacy=0x7f2080137400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207f7fe700 1 -- 172.21.15.112:0/3085202964 >> v1:172.21.15.129:6789/0 conn(0x7f2080098bf0 legacy=0x7f208013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263963752 0 0) 0x7f2080096a60 con 0x7f20800987f0 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient(hunting): my global_id is 4346 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2068003570 con 0x7f20800987f0 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020403346 0 0) 0x7f2080067a50 con 0x7f2080098bf0 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient(hunting): my global_id is 4330 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2080096a60 con 0x7f2080098bf0 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2844530396 0 0) 0x7f2068003570 con 0x7f20800987f0 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 >> v1:172.21.15.129:6789/0 conn(0x7f2080098bf0 legacy=0x7f208013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 >> v1:172.21.15.112:6789/0 conn(0x7f2080067680 legacy=0x7f2080132280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 monclient: found mon.noname-b 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2080096d00 con 0x7f20800987f0 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 5 monclient: authenticate success, global_id 4346 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2074001940 con 0x7f20800987f0 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:27.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: dump: 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2074001bd0 con 0x7f20800987f0 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 1 -- 172.21.15.112:0/3085202964 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2074004a90 con 0x7f20800987f0 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 4 set_mon_vals no callback set 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f207dffb700 10 monclient: dump: 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.188 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 10 monclient: shutdown 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 1 -- 172.21.15.112:0/3085202964 >> v1:172.21.15.112:6790/0 conn(0x7f20800987f0 legacy=0x7f2080137400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 1 -- 172.21.15.112:0/3085202964 shutdown_connections 2024-09-18T11:44:27.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f20858dd700 1 -- 172.21.15.112:0/3085202964 >> 172.21.15.112:0/3085202964 conn(0x7f20800a2010 msgr2=0x7f20800a2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 -- 172.21.15.112:0/3085202964 shutdown_connections 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f1f32cf3700 10 monclient: get_monmap_and_config 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f1f32cf3700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.162+0000 7f1f32cf3700 10 monclient: monmap: 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:27.164514+0000 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:27.164514+0000 2024-09-18T11:44:27.190 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.191 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.191 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.191 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:27.191 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:27.191 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 -- 172.21.15.112:0/3085202964 wait complete. 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient: monmap: 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.192 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.193 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:27.193 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:27.193 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 librados: starting msgr at 2024-09-18T11:44:27.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 librados: starting objecter 2024-09-18T11:44:27.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 Processor -- start 2024-09-18T11:44:27.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f53b0014f70 con 0x7f53b80972f0 2024-09-18T11:44:27.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f539effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f539effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:27.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 -- start start 2024-09-18T11:44:27.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 librados: setting wanted keys 2024-09-18T11:44:27.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 librados: calling monclient init 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient: init 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient(hunting): picked mon.noname-c con 0x7f2080067680 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient(hunting): picked mon.noname-a con 0x7f20800987f0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient(hunting): picked mon.noname-b con 0x7f2080098bf0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20800a0610 con 0x7f20800987f0 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2080128a00 con 0x7f2080098bf0 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2080129160 con 0x7f2080067680 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f20858dd700 10 monclient(hunting): authenticate will time out at 839.010401s 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f207effd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f2080067680 0x7f2080141630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41640/0 (socket says 172.21.15.112:41640) 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.166+0000 7f207effd700 1 -- 172.21.15.112:0/2874692389 learned_addr learned my addr 172.21.15.112:0/2874692389 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f207effd700 1 -- 172.21.15.112:0/2874692389 >> v1:172.21.15.112:6790/0 conn(0x7f2080067680 legacy=0x7f2080141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f207f7fe700 1 -- 172.21.15.112:0/2874692389 >> v1:172.21.15.129:6789/0 conn(0x7f2080098bf0 legacy=0x7f20801499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 client.?.objecter ms_handle_connect 0x7f2080067680 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 client.?.objecter ms_handle_connect 0x7f2080098bf0 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837749964 0 0) 0x7f2080129160 con 0x7f2080067680 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): my global_id is 4349 2024-09-18T11:44:27.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c004f10 con 0x7f2080067680 2024-09-18T11:44:27.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621396016 0 0) 0x7f2080128a00 con 0x7f2080098bf0 2024-09-18T11:44:27.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): my global_id is 4333 2024-09-18T11:44:27.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2080129160 con 0x7f2080098bf0 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f1f32cf3700 1 Processor -- start 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f1f32cf3700 1 -- start start 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f1f32cf3700 10 monclient: init 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 10 monclient(hunting): picked mon.noname-b con 0x7f1f2c131fd0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 10 monclient(hunting): picked mon.noname-c con 0x7f1f2c12edc0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 10 monclient(hunting): picked mon.noname-a con 0x7f1f2c12f8d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c06d060 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c06d4a0 con 0x7f1f2c12edc0 2024-09-18T11:44:27.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c06e680 con 0x7f1f2c131fd0 2024-09-18T11:44:27.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f32cf3700 10 monclient(hunting): authenticate will time out at 839.016764s 2024-09-18T11:44:27.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842660511 0 0) 0x7f206c004f10 con 0x7f2080067680 2024-09-18T11:44:27.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2080128a00 con 0x7f2080067680 2024-09-18T11:44:27.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2078003070 con 0x7f2080067680 2024-09-18T11:44:27.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): dump: 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.201 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.170+0000 7f2066ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1429014126 0 0) 0x7f2080129160 con 0x7f2080098bf0 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f206c004f10 con 0x7f2080098bf0 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2070001c40 con 0x7f2080098bf0 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient(hunting): dump: 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.202 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3290770244 0 0) 0x7f2080128a00 con 0x7f2080067680 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 >> v1:172.21.15.129:6789/0 conn(0x7f2080098bf0 legacy=0x7f20801499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 >> v1:172.21.15.112:6789/0 conn(0x7f20800987f0 legacy=0x7f208012be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 monclient: found mon.c 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208012a050 con 0x7f2080067680 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 5 monclient: authenticate success, global_id 4349 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 10 monclient: _renew_subs 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208012c530 con 0x7f2080067680 2024-09-18T11:44:27.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:27.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 10 monclient: _renew_subs 2024-09-18T11:44:27.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2080142510 con 0x7f2080067680 2024-09-18T11:44:27.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f20858dd700 1 librados: init done 2024-09-18T11:44:27.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f314f0700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f1f2c12edc0 0x7f1f2c12f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.112:36820/0 (socket says 172.21.15.112:36820) 2024-09-18T11:44:27.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f314f0700 1 -- 172.21.15.112:0/1320615478 learned_addr learned my addr 172.21.15.112:0/1320615478 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2078001500 con 0x7f2080067680 2024-09-18T11:44:27.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2078001810 con 0x7f2080067680 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f207cff9700 4 set_mon_vals no callback set 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 monclient: dump: 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.207 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f2078001a20 con 0x7f2080067680 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f2078043b60 con 0x7f2080067680 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 10 client.4349.objecter ms_dispatch 0x7f2080000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 3 client.4349.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 3 client.4349.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f2066ffd700 20 client.4349.objecter dump_active .. 0 homeless 2024-09-18T11:44:27.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f207e7fc700 10 monclient: get_auth_request con 0x7f206c0383a0 auth_method 0 2024-09-18T11:44:27.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f31cf1700 1 -- 172.21.15.112:0/1320615478 >> v1:172.21.15.112:6790/0 conn(0x7f1f2c131fd0 legacy=0x7f1f2c1323a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f324f2700 1 -- 172.21.15.112:0/1320615478 >> v1:172.21.15.112:6789/0 conn(0x7f1f2c12f8d0 legacy=0x7f1f2c13a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.174+0000 7f1f314f0700 1 -- 172.21.15.112:0/1320615478 >> v1:172.21.15.129:6789/0 conn(0x7f1f2c12edc0 legacy=0x7f1f2c12f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.178+0000 7f20858dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.178+0000 7f20858dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.178+0000 7f20858dd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.178+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20600050b0 con 0x7f2080067680 2024-09-18T11:44:27.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.178+0000 7f2066ffd700 10 client.4349.objecter ms_handle_connect 0x7f206c0383a0 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279634893 0 0) 0x7f1f2c06e680 con 0x7f1f2c131fd0 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): my global_id is 4352 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f18003570 con 0x7f1f2c131fd0 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95124851 0 0) 0x7f1f2c06d4a0 con 0x7f1f2c12edc0 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): my global_id is 4336 2024-09-18T11:44:27.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2c06e680 con 0x7f1f2c12edc0 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638777165 0 0) 0x7f1f2c06d060 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient(hunting): my global_id is 4329 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2c06d4a0 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3059124419 0 0) 0x7f1f2c06d4a0 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 >> v1:172.21.15.112:6790/0 conn(0x7f1f2c131fd0 legacy=0x7f1f2c1323a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 >> v1:172.21.15.129:6789/0 conn(0x7f1f2c12edc0 legacy=0x7f1f2c12f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 monclient: found mon.noname-a 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c132ab0 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f32cf3700 5 monclient: authenticate success, global_id 4329 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f32cf3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1f24001b60 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient: dump: 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.216 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f32cf3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 1 -- 172.21.15.112:0/1320615478 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f24001df0 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f30cef700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f32cf3700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:27.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f32cf3700 4 set_mon_vals no callback set 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f32cf3700 10 monclient: shutdown 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f32cf3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.182+0000 7f1f32cf3700 1 -- 172.21.15.112:0/1320615478 >> v1:172.21.15.112:6789/0 conn(0x7f1f2c12f8d0 legacy=0x7f1f2c13a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 -- 172.21.15.112:0/1320615478 shutdown_connections 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 -- 172.21.15.112:0/1320615478 >> 172.21.15.112:0/1320615478 conn(0x7f1f2c067da0 msgr2=0x7f1f2c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 -- 172.21.15.112:0/1320615478 shutdown_connections 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 -- 172.21.15.112:0/1320615478 wait complete. 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient: monmap: 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:27.218 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 librados: starting msgr at 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 librados: starting objecter 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 Processor -- start 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 -- start start 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 librados: setting wanted keys 2024-09-18T11:44:27.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 librados: calling monclient init 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient: init 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient(hunting): picked mon.noname-a con 0x7f1f2c12edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient(hunting): picked mon.noname-b con 0x7f1f2c12f8d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient(hunting): picked mon.noname-c con 0x7f1f2c131fd0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c137c30 con 0x7f1f2c12edc0 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c06ba10 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c06d9d0 con 0x7f1f2c131fd0 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f32cf3700 10 monclient(hunting): authenticate will time out at 839.030382s 2024-09-18T11:44:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f31cf1700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1f2c12edc0 0x7f1f2c1412b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35338/0 (socket says 172.21.15.112:35338) 2024-09-18T11:44:27.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f1f31cf1700 1 -- 172.21.15.112:0/512662696 learned_addr learned my addr 172.21.15.112:0/512662696 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f2078014e00 con 0x7f2080067680 2024-09-18T11:44:27.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f2066ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.186+0000 7f2066ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:27.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f31cf1700 1 -- 172.21.15.112:0/512662696 >> v1:172.21.15.112:6789/0 conn(0x7f1f2c12edc0 legacy=0x7f1f2c1412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 client.?.objecter ms_handle_connect 0x7f1f2c12edc0 2024-09-18T11:44:27.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f324f2700 1 -- 172.21.15.112:0/512662696 >> v1:172.21.15.112:6790/0 conn(0x7f1f2c131fd0 legacy=0x7f1f2c149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 client.?.objecter ms_handle_connect 0x7f1f2c131fd0 2024-09-18T11:44:27.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434624201 0 0) 0x7f1f2c137c30 con 0x7f1f2c12edc0 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): my global_id is 4332 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f10004f10 con 0x7f1f2c12edc0 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 client.?.objecter ms_handle_connect 0x7f1f2c12f8d0 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f314f0700 1 -- 172.21.15.112:0/512662696 >> v1:172.21.15.129:6789/0 conn(0x7f1f2c12f8d0 legacy=0x7f1f2c13afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 954668859 0 0) 0x7f1f10004f10 con 0x7f1f2c12edc0 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f2c137c30 con 0x7f1f2c12edc0 2024-09-18T11:44:27.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416187264 0 0) 0x7f1f2c06ba10 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): my global_id is 4339 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f10004f10 con 0x7f1f2c12f8d0 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1f28001d50 con 0x7f1f2c12edc0 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): dump: 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.227 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 540485788 0 0) 0x7f1f2c137c30 con 0x7f1f2c12edc0 2024-09-18T11:44:27.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 >> v1:172.21.15.112:6790/0 conn(0x7f1f2c131fd0 legacy=0x7f1f2c149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 >> v1:172.21.15.129:6789/0 conn(0x7f1f2c12f8d0 legacy=0x7f1f2c13afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 monclient: found mon.a 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c142150 con 0x7f1f2c12edc0 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 5 monclient: authenticate success, global_id 4332 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 10 monclient: _renew_subs 2024-09-18T11:44:27.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c1391e0 con 0x7f1f2c12edc0 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 10 monclient: _renew_subs 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c138280 con 0x7f1f2c12edc0 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 1 librados: init done 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c09e5e0 con 0x7f1f2c12edc0 2024-09-18T11:44:27.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f280022b0 con 0x7f1f2c12edc0 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f21ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.190+0000 7f1f23fff700 4 set_mon_vals no callback set 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1f28002640 con 0x7f1f2c12edc0 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 10 monclient: dump: 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.231 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f1f28002820 con 0x7f1f2c12edc0 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f1f28043880 con 0x7f1f2c12edc0 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 10 client.4332.objecter ms_dispatch 0x7f1f2c000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 3 client.4332.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:27.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 3 client.4332.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:27.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f21ffb700 20 client.4332.objecter dump_active .. 0 homeless 2024-09-18T11:44:27.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.194+0000 7f1f314f0700 10 monclient: get_auth_request con 0x7f1f100383a0 auth_method 0 2024-09-18T11:44:27.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.198+0000 7f1f21ffb700 10 client.4332.objecter ms_handle_connect 0x7f1f100383a0 2024-09-18T11:44:27.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.202+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f1f28014b80 con 0x7f1f2c12edc0 2024-09-18T11:44:27.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.202+0000 7f1f21ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.202+0000 7f1f21ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:27.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.242+0000 7ff3dd252700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T11:44:27.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.242+0000 7ff3dd252700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T11:44:27.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.242+0000 7ff3dd252700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.242+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff3b8005360 con 0x7ff3d806f800 2024-09-18T11:44:27.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.242+0000 7ff3beffd700 1 -- 172.21.15.112:0/1679192423 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 532354816) 0x7ff3cc0147e0 con 0x7ff3d806f800 2024-09-18T11:44:27.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.242+0000 7ff3beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T11:44:27.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.242+0000 7ff3beffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:27.248 INFO:teuthology.orchestra.run.smithi112.stdout:51539607557 2024-09-18T11:44:27.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 10 librados: watch_flush enter 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 10 librados: watch_flush exit 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 20 client.4331.objecter shutdown clearing up homeless session... 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 10 client.4331.objecter successfully canceled tick 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 >> v1:172.21.15.129:6812/20806 conn(0x7ff3c4038300 legacy=0x7ff3c403a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 10 monclient: shutdown 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 >> v1:172.21.15.112:6790/0 conn(0x7ff3d806f800 legacy=0x7ff3d8145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 shutdown_connections 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 >> 172.21.15.112:0/1679192423 conn(0x7ff3d80a1fb0 msgr2=0x7ff3d8134bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 shutdown_connections 2024-09-18T11:44:27.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 1 -- 172.21.15.112:0/1679192423 wait complete. 2024-09-18T11:44:27.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.246+0000 7ff3dd252700 1 librados: shutdown 2024-09-18T11:44:27.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.2 2024-09-18T11:44:27.262 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:27.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.282+0000 7f4c551ad700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T11:44:27.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.282+0000 7f4c551ad700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T11:44:27.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.282+0000 7f4c551ad700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.282+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4c34005360 con 0x7f4c50067680 2024-09-18T11:44:27.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.282+0000 7f4c4cff9700 1 -- 172.21.15.112:0/98557609 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 532354816) 0x7f4c400145a0 con 0x7f4c50067680 2024-09-18T11:44:27.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.282+0000 7f4c4cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T11:44:27.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.282+0000 7f4c4cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:27.286 INFO:teuthology.orchestra.run.smithi112.stdout:51539607557 2024-09-18T11:44:27.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 10 librados: watch_flush enter 2024-09-18T11:44:27.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 10 librados: watch_flush exit 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 20 client.4308.objecter shutdown clearing up homeless session... 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 10 client.4308.objecter successfully canceled tick 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 >> v1:172.21.15.129:6812/20806 conn(0x7f4c300383f0 legacy=0x7f4c3003a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 10 monclient: shutdown 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 >> v1:172.21.15.112:6789/0 conn(0x7f4c50067680 legacy=0x7f4c5006d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 shutdown_connections 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 >> 172.21.15.112:0/98557609 conn(0x7f4c500a1e30 msgr2=0x7f4c50135bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 shutdown_connections 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 1 -- 172.21.15.112:0/98557609 wait complete. 2024-09-18T11:44:27.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.286+0000 7f4c551ad700 1 librados: shutdown 2024-09-18T11:44:27.303 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-09-18T11:44:27.304 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:27.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T11:44:27.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T11:44:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3cbc005360 con 0x7f3cdc132c60 2024-09-18T11:44:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3cc37fe700 1 -- 172.21.15.112:0/829455225 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 532354816) 0x7f3cd00144c0 con 0x7f3cdc132c60 2024-09-18T11:44:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3cc37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T11:44:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3cc37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:27.333 INFO:teuthology.orchestra.run.smithi112.stdout:51539607557 2024-09-18T11:44:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 10 librados: watch_flush enter 2024-09-18T11:44:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 10 librados: watch_flush exit 2024-09-18T11:44:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 20 client.4337.objecter shutdown clearing up homeless session... 2024-09-18T11:44:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 10 client.4337.objecter successfully canceled tick 2024-09-18T11:44:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 >> v1:172.21.15.129:6812/20806 conn(0x7f3cc80383a0 legacy=0x7f3cc803a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 10 monclient: shutdown 2024-09-18T11:44:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 >> v1:172.21.15.112:6790/0 conn(0x7f3cdc132c60 legacy=0x7f3cdc149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 shutdown_connections 2024-09-18T11:44:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 >> 172.21.15.112:0/829455225 conn(0x7f3cdc0680d0 msgr2=0x7f3cdc133160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 shutdown_connections 2024-09-18T11:44:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 1 -- 172.21.15.112:0/829455225 wait complete. 2024-09-18T11:44:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.330+0000 7f3ce1eb6700 1 librados: shutdown 2024-09-18T11:44:27.346 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.3 2024-09-18T11:44:27.346 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:27.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.398+0000 7f20858dd700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T11:44:27.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.398+0000 7f20858dd700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T11:44:27.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.398+0000 7f20858dd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.398+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f2060005360 con 0x7f2080067680 2024-09-18T11:44:27.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.398+0000 7f2066ffd700 1 -- 172.21.15.112:0/2874692389 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 847495621) 0x7f2078014800 con 0x7f2080067680 2024-09-18T11:44:27.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.398+0000 7f2066ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T11:44:27.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.398+0000 7f2066ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:27.402 INFO:teuthology.orchestra.run.smithi112.stdout:47244640259 2024-09-18T11:44:27.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 10 librados: watch_flush enter 2024-09-18T11:44:27.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 10 librados: watch_flush exit 2024-09-18T11:44:27.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 20 client.4349.objecter shutdown clearing up homeless session... 2024-09-18T11:44:27.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 10 client.4349.objecter successfully canceled tick 2024-09-18T11:44:27.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 >> v1:172.21.15.129:6812/20806 conn(0x7f206c0383a0 legacy=0x7f206c03a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 10 monclient: shutdown 2024-09-18T11:44:27.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 >> v1:172.21.15.112:6790/0 conn(0x7f2080067680 legacy=0x7f2080141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 shutdown_connections 2024-09-18T11:44:27.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 >> 172.21.15.112:0/2874692389 conn(0x7f20800a2010 msgr2=0x7f2080130a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 shutdown_connections 2024-09-18T11:44:27.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 1 -- 172.21.15.112:0/2874692389 wait complete. 2024-09-18T11:44:27.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.402+0000 7f20858dd700 1 librados: shutdown 2024-09-18T11:44:27.419 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2024-09-18T11:44:27.419 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:27.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.442+0000 7f53b7fff700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T11:44:27.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.442+0000 7f53b7fff700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T11:44:27.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.442+0000 7f53b7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.442+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5398005360 con 0x7f53b80972f0 2024-09-18T11:44:27.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.442+0000 7f539effd700 1 -- 172.21.15.112:0/1444722537 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 532354816) 0x7f53b0014970 con 0x7f53b80972f0 2024-09-18T11:44:27.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.442+0000 7f539effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T11:44:27.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.442+0000 7f539effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:27.447 INFO:teuthology.orchestra.run.smithi112.stdout:51539607557 2024-09-18T11:44:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 10 librados: watch_flush enter 2024-09-18T11:44:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 10 librados: watch_flush exit 2024-09-18T11:44:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 20 client.4326.objecter shutdown clearing up homeless session... 2024-09-18T11:44:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 10 client.4326.objecter successfully canceled tick 2024-09-18T11:44:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 >> v1:172.21.15.129:6812/20806 conn(0x7f53a0038a20 legacy=0x7f53a003aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 10 monclient: shutdown 2024-09-18T11:44:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 >> v1:172.21.15.112:6789/0 conn(0x7f53b80972f0 legacy=0x7f53b806d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 shutdown_connections 2024-09-18T11:44:27.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 >> 172.21.15.112:0/1444722537 conn(0x7f53b8067da0 msgr2=0x7f53b80a0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 shutdown_connections 2024-09-18T11:44:27.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 1 -- 172.21.15.112:0/1444722537 wait complete. 2024-09-18T11:44:27.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.446+0000 7f53b7fff700 1 librados: shutdown 2024-09-18T11:44:27.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T11:44:27.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T11:44:27.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1f2c05f2e0 con 0x7f1f2c12edc0 2024-09-18T11:44:27.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f21ffb700 1 -- 172.21.15.112:0/512662696 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 1444633209) 0x7f1f28014580 con 0x7f1f2c12edc0 2024-09-18T11:44:27.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f21ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T11:44:27.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f21ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:27.458 INFO:teuthology.orchestra.run.smithi112.stdout:47244640260 2024-09-18T11:44:27.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-09-18T11:44:27.460 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 10 librados: watch_flush enter 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 10 librados: watch_flush exit 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 20 client.4332.objecter shutdown clearing up homeless session... 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 10 client.4332.objecter successfully canceled tick 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 >> v1:172.21.15.129:6812/20806 conn(0x7f1f100383a0 legacy=0x7f1f1003a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 10 monclient: shutdown 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 >> v1:172.21.15.112:6789/0 conn(0x7f1f2c12edc0 legacy=0x7f1f2c1412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 shutdown_connections 2024-09-18T11:44:27.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 >> 172.21.15.112:0/512662696 conn(0x7f1f2c067da0 msgr2=0x7f1f2c1309d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.454+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 shutdown_connections 2024-09-18T11:44:27.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.458+0000 7f1f32cf3700 1 -- 172.21.15.112:0/512662696 wait complete. 2024-09-18T11:44:27.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.458+0000 7f1f32cf3700 1 librados: shutdown 2024-09-18T11:44:27.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.1 2024-09-18T11:44:27.470 DEBUG:teuthology.parallel:result is None 2024-09-18T11:44:27.470 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T11:44:27.471 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient: get_monmap_and_config 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient: monmap: 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:27.570186+0000 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:27.570186+0000 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.571 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.571 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:27.571 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:27.571 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 1 Processor -- start 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 1 -- start start 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient: init 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient(hunting): picked mon.noname-b con 0x7efeb806df70 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient(hunting): picked mon.noname-c con 0x7efeb806ea50 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient(hunting): picked mon.noname-a con 0x7efeb806d970 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb806b960 con 0x7efeb806d970 2024-09-18T11:44:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb806cac0 con 0x7efeb806ea50 2024-09-18T11:44:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb806c7e0 con 0x7efeb806df70 2024-09-18T11:44:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efebdc21700 10 monclient(hunting): authenticate will time out at 839.411208s 2024-09-18T11:44:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efeb7fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7efeb806d970 0x7efeb8137a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35340/0 (socket says 172.21.15.112:35340) 2024-09-18T11:44:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efeb77fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7efeb806df70 0x7efeb806e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41658/0 (socket says 172.21.15.112:41658) 2024-09-18T11:44:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.566+0000 7efeb7fff700 1 -- 172.21.15.112:0/3065084083 learned_addr learned my addr 172.21.15.112:0/3065084083 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb7fff700 1 -- 172.21.15.112:0/3065084083 >> v1:172.21.15.112:6789/0 conn(0x7efeb806d970 legacy=0x7efeb8137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb77fe700 1 -- 172.21.15.112:0/3065084083 >> v1:172.21.15.112:6790/0 conn(0x7efeb806df70 legacy=0x7efeb806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeaeffd700 1 -- 172.21.15.112:0/3065084083 >> v1:172.21.15.129:6789/0 conn(0x7efeb806ea50 legacy=0x7efeb81342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333701894 0 0) 0x7efeb806b960 con 0x7efeb806d970 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): my global_id is 4335 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea4003570 con 0x7efeb806d970 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465329183 0 0) 0x7efeb806c7e0 con 0x7efeb806df70 2024-09-18T11:44:27.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): my global_id is 4355 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb806b960 con 0x7efeb806df70 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019642986 0 0) 0x7efeb806cac0 con 0x7efeb806ea50 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient(hunting): my global_id is 4342 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb806c7e0 con 0x7efeb806ea50 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950531358 0 0) 0x7efea4003570 con 0x7efeb806d970 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 >> v1:172.21.15.112:6790/0 conn(0x7efeb806df70 legacy=0x7efeb806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 >> v1:172.21.15.129:6789/0 conn(0x7efeb806ea50 legacy=0x7efeb81342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 monclient: found mon.noname-a 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb806cda0 con 0x7efeb806d970 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 5 monclient: authenticate success, global_id 4335 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efeb0001cc0 con 0x7efeb806d970 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: dump: 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeb0001f50 con 0x7efeb806d970 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 4 set_mon_vals no callback set 2024-09-18T11:44:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 1 -- 172.21.15.112:0/3065084083 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efeb0004e10 con 0x7efeb806d970 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efeb6ffd700 10 monclient: dump: 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.580 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 10 monclient: shutdown 2024-09-18T11:44:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 1 -- 172.21.15.112:0/3065084083 >> v1:172.21.15.112:6789/0 conn(0x7efeb806d970 legacy=0x7efeb8137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 1 -- 172.21.15.112:0/3065084083 shutdown_connections 2024-09-18T11:44:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 1 -- 172.21.15.112:0/3065084083 >> 172.21.15.112:0/3065084083 conn(0x7efeb80680d0 msgr2=0x7efeb80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.570+0000 7efebdc21700 1 -- 172.21.15.112:0/3065084083 shutdown_connections 2024-09-18T11:44:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 -- 172.21.15.112:0/3065084083 wait complete. 2024-09-18T11:44:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient: monmap: 2024-09-18T11:44:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 librados: starting msgr at 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 librados: starting objecter 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 Processor -- start 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 -- start start 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 librados: setting wanted keys 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 librados: calling monclient init 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient: init 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient(hunting): picked mon.noname-b con 0x7efeb806d970 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient(hunting): picked mon.noname-c con 0x7efeb806df70 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient(hunting): picked mon.noname-a con 0x7efeb806ea50 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb8120940 con 0x7efeb806ea50 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb81203f0 con 0x7efeb806d970 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb8120660 con 0x7efeb806df70 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efebdc21700 10 monclient(hunting): authenticate will time out at 839.418352s 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efeb7fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7efeb806ea50 0x7efeb8147e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35342/0 (socket says 172.21.15.112:35342) 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.574+0000 7efeb7fff700 1 -- 172.21.15.112:0/3294303801 learned_addr learned my addr 172.21.15.112:0/3294303801 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeb7fff700 1 -- 172.21.15.112:0/3294303801 >> v1:172.21.15.112:6789/0 conn(0x7efeb806ea50 legacy=0x7efeb8147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 client.?.objecter ms_handle_connect 0x7efeb806ea50 2024-09-18T11:44:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 client.?.objecter ms_handle_connect 0x7efeb806df70 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaeffd700 1 -- 172.21.15.112:0/3294303801 >> v1:172.21.15.112:6790/0 conn(0x7efeb806df70 legacy=0x7efeb8144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925480377 0 0) 0x7efeb8120660 con 0x7efeb806df70 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): my global_id is 4358 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeb77fe700 1 -- 172.21.15.112:0/3294303801 >> v1:172.21.15.129:6789/0 conn(0x7efeb806d970 legacy=0x7efeb8131240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe98004f10 con 0x7efeb806df70 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 client.?.objecter ms_handle_connect 0x7efeb806d970 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411459651 0 0) 0x7efeb8120940 con 0x7efeb806ea50 2024-09-18T11:44:27.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): my global_id is 4338 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8120660 con 0x7efeb806ea50 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122395011 0 0) 0x7efeb81203f0 con 0x7efeb806d970 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): my global_id is 4345 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8120940 con 0x7efeb806d970 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180139562 0 0) 0x7efe98004f10 con 0x7efeb806df70 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb81203f0 con 0x7efeb806df70 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2176306101 0 0) 0x7efeb8120660 con 0x7efeb806ea50 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe98004f10 con 0x7efeb806ea50 2024-09-18T11:44:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efea8001c00 con 0x7efeb806df70 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): dump: 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efeb0004d50 con 0x7efeb806ea50 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient(hunting): dump: 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.592 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 343549246 0 0) 0x7efe98004f10 con 0x7efeb806ea50 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 >> v1:172.21.15.112:6790/0 conn(0x7efeb806df70 legacy=0x7efeb8144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 >> v1:172.21.15.129:6789/0 conn(0x7efeb806d970 legacy=0x7efeb8131240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 monclient: found mon.a 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb811f520 con 0x7efeb806ea50 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 5 monclient: authenticate success, global_id 4338 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 10 monclient: _renew_subs 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb8131950 con 0x7efeb806ea50 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 10 monclient: _renew_subs 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb80a0320 con 0x7efeb806ea50 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 1 librados: init done 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeb0005340 con 0x7efeb806ea50 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efeb0005650 con 0x7efeb806ea50 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeb67fc700 4 set_mon_vals no callback set 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 10 monclient: dump: 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7efeb000fed0 con 0x7efeb806ea50 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.578+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c0050b0 con 0x7efeb806ea50 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.582+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7efeb0043b50 con 0x7efeb806ea50 2024-09-18T11:44:27.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.582+0000 7efeaffff700 10 client.4338.objecter ms_dispatch 0x7efeb8000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:27.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.582+0000 7efeaffff700 3 client.4338.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:27.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.582+0000 7efeaffff700 3 client.4338.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:27.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.582+0000 7efeaffff700 20 client.4338.objecter dump_active .. 0 homeless 2024-09-18T11:44:27.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.582+0000 7efeb77fe700 10 monclient: get_auth_request con 0x7efe98038490 auth_method 0 2024-09-18T11:44:27.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.582+0000 7efeaffff700 10 client.4338.objecter ms_handle_connect 0x7efe98038490 2024-09-18T11:44:27.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.590+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7efeb0014f10 con 0x7efeb806ea50 2024-09-18T11:44:27.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.590+0000 7efeaffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.590+0000 7efeaffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:27.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.746+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe9c000b90 con 0x7efe98038490 2024-09-18T11:44:27.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.750+0000 7efeaffff700 1 -- 172.21.15.112:0/3294303801 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41051 (unknown 2965378022 0 3174009328) 0x7efe9c000b90 con 0x7efe98038490 2024-09-18T11:44:27.756 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 10 librados: watch_flush enter 2024-09-18T11:44:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 10 librados: watch_flush exit 2024-09-18T11:44:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 20 client.4338.objecter shutdown clearing up homeless session... 2024-09-18T11:44:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 10 client.4338.objecter successfully canceled tick 2024-09-18T11:44:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 >> v1:172.21.15.129:6812/20806 conn(0x7efe98038490 legacy=0x7efe9803a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 10 monclient: shutdown 2024-09-18T11:44:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 >> v1:172.21.15.112:6789/0 conn(0x7efeb806ea50 legacy=0x7efeb8147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 shutdown_connections 2024-09-18T11:44:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 >> 172.21.15.112:0/3294303801 conn(0x7efeb80680d0 msgr2=0x7efeb809cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 shutdown_connections 2024-09-18T11:44:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 1 -- 172.21.15.112:0/3294303801 wait complete. 2024-09-18T11:44:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.754+0000 7efebdc21700 1 librados: shutdown 2024-09-18T11:44:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T11:44:27.774 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-18T11:44:25.958148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1785164,"kb_used_data":972,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564445876,"statfs":{"total":579820584960,"available":577992577024,"internally_reserved":0,"allocated":995328,"data_stored":1329150,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.761028"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.298671+0000","last_change":"2024-09-18T11:44:24.299141+0000","last_active":"2024-09-18T11:44:24.298671+0000","last_peered":"2024-09-18T11:44:24.298671+0000","last_clean":"2024-09-18T11:44:24.298671+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T11:44:24.298671+0000","last_undegraded":"2024-09-18T11:44:24.298671+0000","last_fullsized":"2024-09-18T11:44:24.298671+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:35:52.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.207977+0000","last_change":"2024-09-18T11:44:24.208317+0000","last_active":"2024-09-18T11:44:24.207977+0000","last_peered":"2024-09-18T11:44:24.207977+0000","last_clean":"2024-09-18T11:44:24.207977+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T11:44:24.207977+0000","last_undegraded":"2024-09-18T11:44:24.207977+0000","last_fullsized":"2024-09-18T11:44:24.207977+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:54.979951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.212606+0000","last_change":"2024-09-18T11:44:24.213224+0000","last_active":"2024-09-18T11:44:24.212606+0000","last_peered":"2024-09-18T11:44:24.212606+0000","last_clean":"2024-09-18T11:44:24.212606+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T11:44:24.212606+0000","last_undegraded":"2024-09-18T11:44:24.212606+0000","last_fullsized":"2024-09-18T11:44:24.212606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:47.310939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.507701+0000","last_change":"2024-09-18T11:44:24.507933+0000","last_active":"2024-09-18T11:44:24.507701+0000","last_peered":"2024-09-18T11:44:24.507701+0000","last_clean":"2024-09-18T11:44:24.507701+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T11:44:24.507701+0000","last_undegraded":"2024-09-18T11:44:24.507701+0000","last_fullsized":"2024-09-18T11:44:24.507701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:56.697013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":101,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.069293+0000","last_change":"2024-09-18T11:44:21.097910+0000","last_active":"2024-09-18T11:44:24.069293+0000","last_peered":"2024-09-18T11:44:24.069293+0000","last_clean":"2024-09-18T11:44:24.069293+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T11:44:24.069293+0000","last_undegraded":"2024-09-18T11:44:24.069293+0000","last_fullsized":"2024-09-18T11:44:24.069293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:48.394073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.507643+0000","last_change":"2024-09-18T11:44:24.508079+0000","last_active":"2024-09-18T11:44:24.507643+0000","last_peered":"2024-09-18T11:44:24.507643+0000","last_clean":"2024-09-18T11:44:24.507643+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T11:44:24.507643+0000","last_undegraded":"2024-09-18T11:44:24.507643+0000","last_fullsized":"2024-09-18T11:44:24.507643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:41:30.648862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.214482+0000","last_change":"2024-09-18T11:44:24.214482+0000","last_active":"2024-09-18T11:44:24.214482+0000","last_peered":"2024-09-18T11:44:24.214482+0000","last_clean":"2024-09-18T11:44:24.214482+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T11:44:24.214482+0000","last_undegraded":"2024-09-18T11:44:24.214482+0000","last_fullsized":"2024-09-18T11:44:24.214482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:26:34.055867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.298527+0000","last_change":"2024-09-18T11:44:24.298975+0000","last_active":"2024-09-18T11:44:24.298527+0000","last_peered":"2024-09-18T11:44:24.298527+0000","last_clean":"2024-09-18T11:44:24.298527+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T11:44:24.298527+0000","last_undegraded":"2024-09-18T11:44:24.298527+0000","last_fullsized":"2024-09-18T11:44:24.298527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:27.175495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.207879+0000","last_change":"2024-09-18T11:44:24.208252+0000","last_active":"2024-09-18T11:44:24.207879+0000","last_peered":"2024-09-18T11:44:24.207879+0000","last_clean":"2024-09-18T11:44:24.207879+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T11:44:24.207879+0000","last_undegraded":"2024-09-18T11:44:24.207879+0000","last_fullsized":"2024-09-18T11:44:24.207879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:08:27.257954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":45379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":44380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]}]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":200704,"data_stored":574307,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":45379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":200704,"data_stored":574307,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":151552,"data_stored":45398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:27.776 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.878+0000 7fdccd6f1700 10 monclient: get_monmap_and_config 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.878+0000 7fdccd6f1700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.878+0000 7fdccd6f1700 10 monclient: monmap: 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:27.883243+0000 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:27.883243+0000 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.883 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.884 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:27.884 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:27.884 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.878+0000 7fdccd6f1700 1 Processor -- start 2024-09-18T11:44:27.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.878+0000 7fdccd6f1700 1 -- start start 2024-09-18T11:44:27.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.878+0000 7fdccd6f1700 10 monclient: init 2024-09-18T11:44:27.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.878+0000 7fdccd6f1700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.878+0000 7fdccd6f1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:27.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 10 monclient(hunting): picked mon.noname-c con 0x7fdcc806e4b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 10 monclient(hunting): picked mon.noname-a con 0x7fdcc809fee0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 10 monclient(hunting): picked mon.noname-b con 0x7fdcc806e880 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc809f300 con 0x7fdcc809fee0 2024-09-18T11:44:27.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc809fca0 con 0x7fdcc806e4b0 2024-09-18T11:44:27.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc8096be0 con 0x7fdcc806e880 2024-09-18T11:44:27.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 10 monclient(hunting): authenticate will time out at 839.724119s 2024-09-18T11:44:27.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc77fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fdcc806e880 0x7fdcc8138510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41676/0 (socket says 172.21.15.112:41676) 2024-09-18T11:44:27.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc77fe700 1 -- 172.21.15.112:0/2894726747 learned_addr learned my addr 172.21.15.112:0/2894726747 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc77fe700 1 -- 172.21.15.112:0/2894726747 >> v1:172.21.15.112:6790/0 conn(0x7fdcc806e880 legacy=0x7fdcc8138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc67fc700 1 -- 172.21.15.112:0/2894726747 >> v1:172.21.15.112:6789/0 conn(0x7fdcc809fee0 legacy=0x7fdcc809ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc6ffd700 1 -- 172.21.15.112:0/2894726747 >> v1:172.21.15.129:6789/0 conn(0x7fdcc806e4b0 legacy=0x7fdcc809e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812895312 0 0) 0x7fdcc8096be0 con 0x7fdcc806e880 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): my global_id is 4361 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb0003570 con 0x7fdcc806e880 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595306461 0 0) 0x7fdcc809f300 con 0x7fdcc809fee0 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): my global_id is 4341 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc8096be0 con 0x7fdcc809fee0 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047670369 0 0) 0x7fdcc809fca0 con 0x7fdcc806e4b0 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient(hunting): my global_id is 4348 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc809f300 con 0x7fdcc806e4b0 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3324415444 0 0) 0x7fdcb0003570 con 0x7fdcc806e880 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 >> v1:172.21.15.129:6789/0 conn(0x7fdcc806e4b0 legacy=0x7fdcc809e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 >> v1:172.21.15.112:6789/0 conn(0x7fdcc809fee0 legacy=0x7fdcc809ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 monclient: found mon.noname-b 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc806d6f0 con 0x7fdcc806e880 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 5 monclient: authenticate success, global_id 4361 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdcb8001940 con 0x7fdcc806e880 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: dump: 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.892 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcb8001bd0 con 0x7fdcc806e880 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 1 -- 172.21.15.112:0/2894726747 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdcb8004a90 con 0x7fdcc806e880 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdccd6f1700 4 set_mon_vals no callback set 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.882+0000 7fdcc5ffb700 10 monclient: dump: 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.894 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 10 monclient: shutdown 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 -- 172.21.15.112:0/2894726747 >> v1:172.21.15.112:6790/0 conn(0x7fdcc806e880 legacy=0x7fdcc8138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 -- 172.21.15.112:0/2894726747 shutdown_connections 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 -- 172.21.15.112:0/2894726747 >> 172.21.15.112:0/2894726747 conn(0x7fdcc8068100 msgr2=0x7fdcc80a3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 -- 172.21.15.112:0/2894726747 shutdown_connections 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 -- 172.21.15.112:0/2894726747 wait complete. 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 10 monclient: build_initial_monmap 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 10 monclient: monmap: 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:27.895 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:27.896 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:27.896 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:27.896 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.896 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:27.896 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:27.896 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:27.896 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 librados: starting msgr at 2024-09-18T11:44:27.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 librados: starting objecter 2024-09-18T11:44:27.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 Processor -- start 2024-09-18T11:44:27.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 -- start start 2024-09-18T11:44:27.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 librados: setting wanted keys 2024-09-18T11:44:27.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 1 librados: calling monclient init 2024-09-18T11:44:27.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.886+0000 7fdccd6f1700 10 monclient: init 2024-09-18T11:44:27.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient(hunting): picked mon.noname-c con 0x7fdcc806e4b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient(hunting): picked mon.noname-a con 0x7fdcc806e880 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient(hunting): picked mon.noname-b con 0x7fdcc809fee0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc809d920 con 0x7fdcc806e880 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc81223c0 con 0x7fdcc809fee0 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc81226a0 con 0x7fdcc806e4b0 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcc67fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdcc806e880 0x7fdcc81392e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35366/0 (socket says 172.21.15.112:35366) 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient(hunting): authenticate will time out at 839.732368s 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcc67fc700 1 -- 172.21.15.112:0/95086665 learned_addr learned my addr 172.21.15.112:0/95086665 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:27.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcc6ffd700 1 -- 172.21.15.112:0/95086665 >> v1:172.21.15.112:6790/0 conn(0x7fdcc806e4b0 legacy=0x7fdcc8141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 client.?.objecter ms_handle_connect 0x7fdcc806e4b0 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 client.?.objecter ms_handle_connect 0x7fdcc806e880 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcc67fc700 1 -- 172.21.15.112:0/95086665 >> v1:172.21.15.112:6789/0 conn(0x7fdcc806e880 legacy=0x7fdcc81392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941107584 0 0) 0x7fdcc81226a0 con 0x7fdcc806e4b0 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): my global_id is 4364 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcc77fe700 1 -- 172.21.15.112:0/95086665 >> v1:172.21.15.129:6789/0 conn(0x7fdcc809fee0 legacy=0x7fdcc81499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb4004f10 con 0x7fdcc806e4b0 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 client.?.objecter ms_handle_connect 0x7fdcc809fee0 2024-09-18T11:44:27.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329594669 0 0) 0x7fdcc809d920 con 0x7fdcc806e880 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): my global_id is 4344 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc81226a0 con 0x7fdcc806e880 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591639183 0 0) 0x7fdcc81223c0 con 0x7fdcc809fee0 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): my global_id is 4351 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc809d920 con 0x7fdcc809fee0 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1542967254 0 0) 0x7fdcb4004f10 con 0x7fdcc806e4b0 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc81223c0 con 0x7fdcc806e4b0 2024-09-18T11:44:27.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdcc0001f20 con 0x7fdcc806e4b0 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): dump: 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3902826055 0 0) 0x7fdcc81226a0 con 0x7fdcc806e880 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcb4004f10 con 0x7fdcc806e880 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdcbc001f00 con 0x7fdcc806e880 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient(hunting): dump: 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.906 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2108272109 0 0) 0x7fdcc81223c0 con 0x7fdcc806e4b0 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 >> v1:172.21.15.129:6789/0 conn(0x7fdcc809fee0 legacy=0x7fdcc81499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 >> v1:172.21.15.112:6789/0 conn(0x7fdcc806e880 legacy=0x7fdcc81392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 monclient: found mon.c 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc8141d30 con 0x7fdcc806e4b0 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdcaeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:27.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 5 monclient: authenticate success, global_id 4364 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient: _renew_subs 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.890+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc81399f0 con 0x7fdcc806e4b0 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcc0002530 con 0x7fdcc806e4b0 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 10 monclient: _renew_subs 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdcc0002840 con 0x7fdcc806e4b0 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc8121670 con 0x7fdcc806e4b0 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 1 librados: init done 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:27.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 10 monclient: dump: 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcc57fa700 4 set_mon_vals no callback set 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fdcc0002a50 con 0x7fdcc806e4b0 2024-09-18T11:44:27.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc805f2e0 con 0x7fdcc806e4b0 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7fdcc0043960 con 0x7fdcc806e4b0 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 10 client.4364.objecter ms_dispatch 0x7fdcc8000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 3 client.4364.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 3 client.4364.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcaeffd700 20 client.4364.objecter dump_active .. 0 homeless 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.894+0000 7fdcc67fc700 10 monclient: get_auth_request con 0x7fdcb4038430 auth_method 0 2024-09-18T11:44:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.898+0000 7fdcaeffd700 10 client.4364.objecter ms_handle_connect 0x7fdcb4038430 2024-09-18T11:44:27.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.902+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fdcc0002d00 con 0x7fdcc806e4b0 2024-09-18T11:44:27.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.902+0000 7fdcaeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:27.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:27.902+0000 7fdcaeffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:28.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.062+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdcc8142570 con 0x7fdcb4038430 2024-09-18T11:44:28.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.066+0000 7fdcaeffd700 1 -- 172.21.15.112:0/95086665 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41051 (unknown 2965378022 0 1276846444) 0x7fdcc8142570 con 0x7fdcb4038430 2024-09-18T11:44:28.071 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:28.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 10 librados: watch_flush enter 2024-09-18T11:44:28.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 10 librados: watch_flush exit 2024-09-18T11:44:28.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 20 client.4364.objecter shutdown clearing up homeless session... 2024-09-18T11:44:28.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 10 client.4364.objecter successfully canceled tick 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 >> v1:172.21.15.129:6812/20806 conn(0x7fdcb4038430 legacy=0x7fdcb403a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 10 monclient: shutdown 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 >> v1:172.21.15.112:6790/0 conn(0x7fdcc806e4b0 legacy=0x7fdcc8141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 shutdown_connections 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 >> 172.21.15.112:0/95086665 conn(0x7fdcc8068100 msgr2=0x7fdcc80a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 shutdown_connections 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 1 -- 172.21.15.112:0/95086665 wait complete. 2024-09-18T11:44:28.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.070+0000 7fdccd6f1700 1 librados: shutdown 2024-09-18T11:44:28.075 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T11:44:28.085 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-18T11:44:27.958730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1785164,"kb_used_data":972,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564445876,"statfs":{"total":579820584960,"available":577992577024,"internally_reserved":0,"allocated":995328,"data_stored":1329150,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.761610"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.298671+0000","last_change":"2024-09-18T11:44:24.299141+0000","last_active":"2024-09-18T11:44:24.298671+0000","last_peered":"2024-09-18T11:44:24.298671+0000","last_clean":"2024-09-18T11:44:24.298671+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T11:44:24.298671+0000","last_undegraded":"2024-09-18T11:44:24.298671+0000","last_fullsized":"2024-09-18T11:44:24.298671+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:35:52.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.207977+0000","last_change":"2024-09-18T11:44:24.208317+0000","last_active":"2024-09-18T11:44:24.207977+0000","last_peered":"2024-09-18T11:44:24.207977+0000","last_clean":"2024-09-18T11:44:24.207977+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T11:44:24.207977+0000","last_undegraded":"2024-09-18T11:44:24.207977+0000","last_fullsized":"2024-09-18T11:44:24.207977+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:54.979951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.212606+0000","last_change":"2024-09-18T11:44:24.213224+0000","last_active":"2024-09-18T11:44:24.212606+0000","last_peered":"2024-09-18T11:44:24.212606+0000","last_clean":"2024-09-18T11:44:24.212606+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T11:44:24.212606+0000","last_undegraded":"2024-09-18T11:44:24.212606+0000","last_fullsized":"2024-09-18T11:44:24.212606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:47.310939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.507701+0000","last_change":"2024-09-18T11:44:24.507933+0000","last_active":"2024-09-18T11:44:24.507701+0000","last_peered":"2024-09-18T11:44:24.507701+0000","last_clean":"2024-09-18T11:44:24.507701+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T11:44:24.507701+0000","last_undegraded":"2024-09-18T11:44:24.507701+0000","last_fullsized":"2024-09-18T11:44:24.507701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:56.697013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":101,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.069293+0000","last_change":"2024-09-18T11:44:21.097910+0000","last_active":"2024-09-18T11:44:24.069293+0000","last_peered":"2024-09-18T11:44:24.069293+0000","last_clean":"2024-09-18T11:44:24.069293+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T11:44:24.069293+0000","last_undegraded":"2024-09-18T11:44:24.069293+0000","last_fullsized":"2024-09-18T11:44:24.069293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:48.394073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.507643+0000","last_change":"2024-09-18T11:44:24.508079+0000","last_active":"2024-09-18T11:44:24.507643+0000","last_peered":"2024-09-18T11:44:24.507643+0000","last_clean":"2024-09-18T11:44:24.507643+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T11:44:24.507643+0000","last_undegraded":"2024-09-18T11:44:24.507643+0000","last_fullsized":"2024-09-18T11:44:24.507643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:41:30.648862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.214482+0000","last_change":"2024-09-18T11:44:24.214482+0000","last_active":"2024-09-18T11:44:24.214482+0000","last_peered":"2024-09-18T11:44:24.214482+0000","last_clean":"2024-09-18T11:44:24.214482+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T11:44:24.214482+0000","last_undegraded":"2024-09-18T11:44:24.214482+0000","last_fullsized":"2024-09-18T11:44:24.214482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:26:34.055867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.298527+0000","last_change":"2024-09-18T11:44:24.298975+0000","last_active":"2024-09-18T11:44:24.298527+0000","last_peered":"2024-09-18T11:44:24.298527+0000","last_clean":"2024-09-18T11:44:24.298527+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T11:44:24.298527+0000","last_undegraded":"2024-09-18T11:44:24.298527+0000","last_fullsized":"2024-09-18T11:44:24.298527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:27.175495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.207879+0000","last_change":"2024-09-18T11:44:24.208252+0000","last_active":"2024-09-18T11:44:24.207879+0000","last_peered":"2024-09-18T11:44:24.207879+0000","last_clean":"2024-09-18T11:44:24.207879+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T11:44:24.207879+0000","last_undegraded":"2024-09-18T11:44:24.207879+0000","last_fullsized":"2024-09-18T11:44:24.207879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:08:27.257954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":45379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":44380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]}]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":200704,"data_stored":574307,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":45379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":200704,"data_stored":574307,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":151552,"data_stored":45398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:28.086 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T11:44:28.086 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T11:44:28.087 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-18T11:44:28.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-18T11:44:28.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient: get_monmap_and_config 2024-09-18T11:44:28.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient: build_initial_monmap 2024-09-18T11:44:28.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient: monmap: 2024-09-18T11:44:28.190 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:28.190 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.190 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:28.190290+0000 2024-09-18T11:44:28.191 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:28.190290+0000 2024-09-18T11:44:28.191 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:28.191 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.191 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:28.191 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:28.191 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:28.191 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 1 Processor -- start 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 1 -- start start 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient: init 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient(hunting): picked mon.noname-c con 0x7f88300a21c0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient(hunting): picked mon.noname-b con 0x7f883009fdf0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient(hunting): picked mon.noname-a con 0x7f88300a01f0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883006b7c0 con 0x7f88300a01f0 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88300a2590 con 0x7f88300a21c0 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883009beb0 con 0x7f883009fdf0 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f8834b3c700 10 monclient(hunting): authenticate will time out at 840.031139s 2024-09-18T11:44:28.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f882ed9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f88300a01f0 0x7f883013f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35378/0 (socket says 172.21.15.112:35378) 2024-09-18T11:44:28.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.186+0000 7f882ed9d700 1 -- 172.21.15.112:0/991039017 learned_addr learned my addr 172.21.15.112:0/991039017 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:28.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882ed9d700 1 -- 172.21.15.112:0/991039017 >> v1:172.21.15.112:6789/0 conn(0x7f88300a01f0 legacy=0x7f883013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882dd9b700 1 -- 172.21.15.112:0/991039017 >> v1:172.21.15.112:6790/0 conn(0x7f883009fdf0 legacy=0x7f883013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882e59c700 1 -- 172.21.15.112:0/991039017 >> v1:172.21.15.129:6789/0 conn(0x7f88300a21c0 legacy=0x7f883009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799416969 0 0) 0x7f883009beb0 con 0x7f883009fdf0 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): my global_id is 4367 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8818003570 con 0x7f883009fdf0 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249216264 0 0) 0x7f883006b7c0 con 0x7f88300a01f0 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): my global_id is 4347 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f883009beb0 con 0x7f88300a01f0 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145608687 0 0) 0x7f88300a2590 con 0x7f88300a21c0 2024-09-18T11:44:28.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient(hunting): my global_id is 4354 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f883006b7c0 con 0x7f88300a21c0 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1184502583 0 0) 0x7f8818003570 con 0x7f883009fdf0 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 >> v1:172.21.15.129:6789/0 conn(0x7f88300a21c0 legacy=0x7f883009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 >> v1:172.21.15.112:6789/0 conn(0x7f88300a01f0 legacy=0x7f883013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: _finish_hunting 0 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 monclient: found mon.noname-b 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88300a0fb0 con 0x7f883009fdf0 2024-09-18T11:44:28.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: _finish_auth 0 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8820001dd0 con 0x7f883009fdf0 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: dump: 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.199 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 5 monclient: authenticate success, global_id 4367 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8820001fb0 con 0x7f883009fdf0 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 4 set_mon_vals no callback set 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 1 -- 172.21.15.112:0/991039017 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8820002420 con 0x7f883009fdf0 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f882d59a700 10 monclient: dump: 2024-09-18T11:44:28.200 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 10 monclient: shutdown 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:28.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 1 -- 172.21.15.112:0/991039017 >> v1:172.21.15.112:6790/0 conn(0x7f883009fdf0 legacy=0x7f883013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 1 -- 172.21.15.112:0/991039017 shutdown_connections 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 1 -- 172.21.15.112:0/991039017 >> 172.21.15.112:0/991039017 conn(0x7f88300680d0 msgr2=0x7f88300a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 1 -- 172.21.15.112:0/991039017 shutdown_connections 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.190+0000 7f8834b3c700 1 -- 172.21.15.112:0/991039017 wait complete. 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient: build_initial_monmap 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient: monmap: 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.202 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:28.203 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:28.203 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:28.203 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 librados: starting msgr at 2024-09-18T11:44:28.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 librados: starting objecter 2024-09-18T11:44:28.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 Processor -- start 2024-09-18T11:44:28.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 -- start start 2024-09-18T11:44:28.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 librados: setting wanted keys 2024-09-18T11:44:28.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 librados: calling monclient init 2024-09-18T11:44:28.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient: init 2024-09-18T11:44:28.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:28.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient(hunting): picked mon.noname-c con 0x7f883009fdf0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient(hunting): picked mon.noname-b con 0x7f88300a01f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient(hunting): picked mon.noname-a con 0x7f88300a21c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8830125520 con 0x7f88300a21c0 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883006edd0 con 0x7f88300a01f0 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883009d840 con 0x7f883009fdf0 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f8834b3c700 10 monclient(hunting): authenticate will time out at 840.037837s 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f882ed9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f88300a21c0 0x7f883014e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35394/0 (socket says 172.21.15.112:35394) 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.194+0000 7f882ed9d700 1 -- 172.21.15.112:0/412490364 learned_addr learned my addr 172.21.15.112:0/412490364 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f882ed9d700 1 -- 172.21.15.112:0/412490364 >> v1:172.21.15.112:6789/0 conn(0x7f88300a21c0 legacy=0x7f883014e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 client.?.objecter ms_handle_connect 0x7f88300a21c0 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 client.?.objecter ms_handle_connect 0x7f883009fdf0 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f882e59c700 1 -- 172.21.15.112:0/412490364 >> v1:172.21.15.112:6790/0 conn(0x7f883009fdf0 legacy=0x7f88301399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f882dd9b700 1 -- 172.21.15.112:0/412490364 >> v1:172.21.15.129:6789/0 conn(0x7f88300a01f0 legacy=0x7f883014ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736994650 0 0) 0x7f8830125520 con 0x7f88300a21c0 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): my global_id is 4350 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8810004f10 con 0x7f88300a21c0 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 client.?.objecter ms_handle_connect 0x7f88300a01f0 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122980824 0 0) 0x7f883009d840 con 0x7f883009fdf0 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): my global_id is 4370 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8830125520 con 0x7f883009fdf0 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473210584 0 0) 0x7f883006edd0 con 0x7f88300a01f0 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): my global_id is 4357 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f883009d840 con 0x7f88300a01f0 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 66925444 0 0) 0x7f8810004f10 con 0x7f88300a21c0 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f883006edd0 con 0x7f88300a21c0 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8828001c20 con 0x7f88300a21c0 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:28.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3503385027 0 0) 0x7f8830125520 con 0x7f883009fdf0 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8810004f10 con 0x7f883009fdf0 2024-09-18T11:44:28.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8824001bd0 con 0x7f883009fdf0 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.212 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3858818195 0 0) 0x7f883006edd0 con 0x7f88300a21c0 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 >> v1:172.21.15.112:6790/0 conn(0x7f883009fdf0 legacy=0x7f88301399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 >> v1:172.21.15.129:6789/0 conn(0x7f88300a01f0 legacy=0x7f883014ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 monclient: found mon.a 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8830125b70 con 0x7f88300a21c0 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 5 monclient: authenticate success, global_id 4350 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 10 monclient: _renew_subs 2024-09-18T11:44:28.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8830124fa0 con 0x7f88300a21c0 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 10 monclient: _renew_subs 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f883013a0c0 con 0x7f88300a21c0 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 1 librados: init done 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8828004be0 con 0x7f88300a21c0 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f882cd99700 4 set_mon_vals no callback set 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8828004fb0 con 0x7f88300a21c0 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 monclient: dump: 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88140050b0 con 0x7f88300a21c0 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8828005110 con 0x7f88300a21c0 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3558+0+0 (unknown 1872962265 0 0) 0x7f8828043830 con 0x7f88300a21c0 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 10 client.4350.objecter ms_dispatch 0x7f8830000fd0 osd_map(17..17 src has 1..17) v4 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 3 client.4350.objecter handle_osd_map got epochs [17,17] > 0 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.198+0000 7f881e7fc700 3 client.4350.objecter handle_osd_map decoding full epoch 17 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.202+0000 7f881e7fc700 20 client.4350.objecter dump_active .. 0 homeless 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.202+0000 7f882e59c700 10 monclient: get_auth_request con 0x7f8810038270 auth_method 0 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.202+0000 7f881e7fc700 10 client.4350.objecter ms_handle_connect 0x7f8810038270 2024-09-18T11:44:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.210+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f88280053c0 con 0x7f88300a21c0 2024-09-18T11:44:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.210+0000 7f881e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.210+0000 7f881e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:28.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f8834b3c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-18T11:44:28.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f8834b3c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-18T11:44:28.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f8834b3c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f88140058e0 con 0x7f88300a21c0 2024-09-18T11:44:28.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f881e7fc700 1 -- 172.21.15.112:0/412490364 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f8828014530 con 0x7f88300a21c0 2024-09-18T11:44:28.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f881e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-18T11:44:28.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f881e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:28.415 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:28.415 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T11:44:28.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f8834b3c700 10 librados: watch_flush enter 2024-09-18T11:44:28.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f8834b3c700 10 librados: watch_flush exit 2024-09-18T11:44:28.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f8834b3c700 20 client.4350.objecter shutdown clearing up homeless session... 2024-09-18T11:44:28.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.410+0000 7f8834b3c700 10 client.4350.objecter successfully canceled tick 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 >> v1:172.21.15.129:6812/20806 conn(0x7f8810038270 legacy=0x7f881003a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 10 monclient: shutdown 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 >> v1:172.21.15.112:6789/0 conn(0x7f88300a21c0 legacy=0x7f883014e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 shutdown_connections 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 >> 172.21.15.112:0/412490364 conn(0x7f88300680d0 msgr2=0x7f883009c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 shutdown_connections 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 1 -- 172.21.15.112:0/412490364 wait complete. 2024-09-18T11:44:28.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.414+0000 7f8834b3c700 1 librados: shutdown 2024-09-18T11:44:28.428 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-18T11:44:28.428 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-18T11:44:28.442 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-18T11:44:28.442 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:44:28.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient: get_monmap_and_config 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient: build_initial_monmap 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient: monmap: 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:28.548869+0000 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:28.548869+0000 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.549 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:28.550 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:28.550 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:28.550 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 1 Processor -- start 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 1 -- start start 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient: init 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient(hunting): picked mon.noname-a con 0x7fdefc0a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient(hunting): picked mon.noname-b con 0x7fdefc096a60 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient(hunting): picked mon.noname-c con 0x7fdefc13ca20 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc06e910 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc06e630 con 0x7fdefc13ca20 2024-09-18T11:44:28.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc0a2270 con 0x7fdefc096a60 2024-09-18T11:44:28.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:28.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf02df0700 10 monclient(hunting): authenticate will time out at 840.390113s 2024-09-18T11:44:28.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf01dee700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdefc0a1ea0 0x7fdefc06bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35404/0 (socket says 172.21.15.112:35404) 2024-09-18T11:44:28.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.546+0000 7fdf01dee700 1 -- 172.21.15.112:0/3603762673 learned_addr learned my addr 172.21.15.112:0/3603762673 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf01dee700 1 -- 172.21.15.112:0/3603762673 >> v1:172.21.15.112:6789/0 conn(0x7fdefc0a1ea0 legacy=0x7fdefc06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf015ed700 1 -- 172.21.15.112:0/3603762673 >> v1:172.21.15.112:6790/0 conn(0x7fdefc096a60 legacy=0x7fdefc06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf025ef700 1 -- 172.21.15.112:0/3603762673 >> v1:172.21.15.129:6789/0 conn(0x7fdefc13ca20 legacy=0x7fdefc13ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343360998 0 0) 0x7fdefc06e910 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): my global_id is 4353 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee8003570 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53189618 0 0) 0x7fdefc0a2270 con 0x7fdefc096a60 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): my global_id is 4373 2024-09-18T11:44:28.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdefc06e910 con 0x7fdefc096a60 2024-09-18T11:44:28.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752215434 0 0) 0x7fdefc06e630 con 0x7fdefc13ca20 2024-09-18T11:44:28.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient(hunting): my global_id is 4360 2024-09-18T11:44:28.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdefc0a2270 con 0x7fdefc13ca20 2024-09-18T11:44:28.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278175515 0 0) 0x7fdee8003570 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 >> v1:172.21.15.112:6790/0 conn(0x7fdefc096a60 legacy=0x7fdefc06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 >> v1:172.21.15.129:6789/0 conn(0x7fdefc13ca20 legacy=0x7fdefc13ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: _finish_hunting 0 2024-09-18T11:44:28.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 monclient: found mon.noname-a 2024-09-18T11:44:28.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:28.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc06eed0 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: _finish_auth 0 2024-09-18T11:44:28.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:28.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdef4001cc0 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:28.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: dump: 2024-09-18T11:44:28.558 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.558 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.558 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.559 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.559 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.559 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.559 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.559 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.559 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.560 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 5 monclient: authenticate success, global_id 4353 2024-09-18T11:44:28.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:28.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdef4001f50 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 1 -- 172.21.15.112:0/3603762673 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdef4004e10 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf00dec700 10 monclient: dump: 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.561 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.562 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.562 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.562 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.562 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:28.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 4 set_mon_vals no callback set 2024-09-18T11:44:28.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 10 monclient: shutdown 2024-09-18T11:44:28.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 1 -- 172.21.15.112:0/3603762673 >> v1:172.21.15.112:6789/0 conn(0x7fdefc0a1ea0 legacy=0x7fdefc06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 1 -- 172.21.15.112:0/3603762673 shutdown_connections 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 1 -- 172.21.15.112:0/3603762673 >> 172.21.15.112:0/3603762673 conn(0x7fdefc067da0 msgr2=0x7fdefc0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 1 -- 172.21.15.112:0/3603762673 shutdown_connections 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 1 -- 172.21.15.112:0/3603762673 wait complete. 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 10 monclient: build_initial_monmap 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.550+0000 7fdf02df0700 10 monclient: monmap: 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:28.563 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 librados: starting msgr at 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 librados: starting objecter 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 Processor -- start 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 -- start start 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 librados: setting wanted keys 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 librados: calling monclient init 2024-09-18T11:44:28.564 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 10 monclient: init 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 10 monclient(hunting): picked mon.noname-c con 0x7fdefc096a60 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 10 monclient(hunting): picked mon.noname-a con 0x7fdefc0a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 10 monclient(hunting): picked mon.noname-b con 0x7fdefc13ca20 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc06e110 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc09dc80 con 0x7fdefc13ca20 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc126880 con 0x7fdefc096a60 2024-09-18T11:44:28.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf02df0700 10 monclient(hunting): authenticate will time out at 840.397411s 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf015ed700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdefc0a1ea0 0x7fdefc123840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35412/0 (socket says 172.21.15.112:35412) 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf015ed700 1 -- 172.21.15.112:0/2866763170 learned_addr learned my addr 172.21.15.112:0/2866763170 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf01dee700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fdefc096a60 0x7fdefc123130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41698/0 (socket says 172.21.15.112:41698) 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf015ed700 1 -- 172.21.15.112:0/2866763170 >> v1:172.21.15.112:6789/0 conn(0x7fdefc0a1ea0 legacy=0x7fdefc123840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 10 client.?.objecter ms_handle_connect 0x7fdefc0a1ea0 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 10 client.?.objecter ms_handle_connect 0x7fdefc096a60 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdf01dee700 1 -- 172.21.15.112:0/2866763170 >> v1:172.21.15.112:6790/0 conn(0x7fdefc096a60 legacy=0x7fdefc123130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115761208 0 0) 0x7fdefc06e110 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 10 monclient(hunting): my global_id is 4356 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.554+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee0004f10 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561798159 0 0) 0x7fdefc126880 con 0x7fdefc096a60 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): my global_id is 4376 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdefc06e110 con 0x7fdefc096a60 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 client.?.objecter ms_handle_connect 0x7fdefc13ca20 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf025ef700 1 -- 172.21.15.112:0/2866763170 >> v1:172.21.15.129:6789/0 conn(0x7fdefc13ca20 legacy=0x7fdefc14ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569957985 0 0) 0x7fdefc09dc80 con 0x7fdefc13ca20 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): my global_id is 4363 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdefc126880 con 0x7fdefc13ca20 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2780680237 0 0) 0x7fdee0004f10 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdefc09dc80 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdeec001f00 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): dump: 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.571 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571095940 0 0) 0x7fdefc06e110 con 0x7fdefc096a60 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdee0004f10 con 0x7fdefc096a60 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdef4004bb0 con 0x7fdefc096a60 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient(hunting): dump: 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.572 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 53294155 0 0) 0x7fdefc09dc80 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 >> v1:172.21.15.112:6790/0 conn(0x7fdefc096a60 legacy=0x7fdefc123130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 >> v1:172.21.15.129:6789/0 conn(0x7fdefc13ca20 legacy=0x7fdefc14ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 monclient: found mon.a 2024-09-18T11:44:28.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc124ae0 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 5 monclient: authenticate success, global_id 4356 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 10 monclient: _renew_subs 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdefc1265a0 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 10 monclient: _renew_subs 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdefc13c690 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 1 librados: init done 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdeec002310 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdeec002660 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef3fff700 4 set_mon_vals no callback set 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 monclient: dump: 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.575 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fdeec002870 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdedc0050b0 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fdeec043830 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 10 client.4356.objecter ms_dispatch 0x7fdefc000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:28.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 3 client.4356.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:28.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 3 client.4356.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:28.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdef1ffb700 20 client.4356.objecter dump_active .. 0 homeless 2024-09-18T11:44:28.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.558+0000 7fdf01dee700 10 monclient: get_auth_request con 0x7fdee0038200 auth_method 0 2024-09-18T11:44:28.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.562+0000 7fdef1ffb700 10 client.4356.objecter ms_handle_connect 0x7fdee0038200 2024-09-18T11:44:28.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.566+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fdeec014b40 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.566+0000 7fdef1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.566+0000 7fdef1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:28.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.730+0000 7fdf02df0700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:28.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.730+0000 7fdf02df0700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:28.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.730+0000 7fdf02df0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:28.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.730+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdedc005360 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.730+0000 7fdef1ffb700 1 -- 172.21.15.112:0/2866763170 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+10065 (unknown 3338201251 0 3138079410) 0x7fdeec014540 con 0x7fdefc0a1ea0 2024-09-18T11:44:28.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.730+0000 7fdef1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T11:44:28.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.730+0000 7fdef1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:28.736 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:28.736 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":18,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","created":"2024-09-18T11:44:09.317663+0000","modified":"2024-09-18T11:44:28.212904+0000","last_up_change":"2024-09-18T11:44:19.034653+0000","last_in_change":"2024-09-18T11:44:11.615001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:44:19.901214+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-18T11:44:20.029398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ecb0abea-5208-453b-86e2-58b64b0634ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":20984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":20984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":20984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":20984}]},"public_addr":"172.21.15.112:6804/20984","cluster_addr":"172.21.15.112:6805/20984","heartbeat_back_addr":"172.21.15.112:6807/20984","heartbeat_front_addr":"172.21.15.112:6806/20984","state":["exists","up"]},{"osd":1,"uuid":"3164789a-58af-4cf7-bd86-260d70dd02c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":20985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":20985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":20985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":20985}]},"public_addr":"172.21.15.112:6800/20985","cluster_addr":"172.21.15.112:6801/20985","heartbeat_back_addr":"172.21.15.112:6803/20985","heartbeat_front_addr":"172.21.15.112:6802/20985","state":["exists","up"]},{"osd":2,"uuid":"0223fabc-71f3-4642-8818-d99dbd5ba501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":20986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":20986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":20986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":20986}]},"public_addr":"172.21.15.112:6808/20986","cluster_addr":"172.21.15.112:6809/20986","heartbeat_back_addr":"172.21.15.112:6811/20986","heartbeat_front_addr":"172.21.15.112:6810/20986","state":["exists","up"]},{"osd":3,"uuid":"07461389-250c-4e30-8890-4267bddeac4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6804","nonce":21049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6805","nonce":21049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6807","nonce":21049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6806","nonce":21049}]},"public_addr":"172.21.15.129:6804/21049","cluster_addr":"172.21.15.129:6805/21049","heartbeat_back_addr":"172.21.15.129:6807/21049","heartbeat_front_addr":"172.21.15.129:6806/21049","state":["exists","up"]},{"osd":4,"uuid":"ad3a24d7-0278-4137-a499-e7b7337e1698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6808","nonce":21047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6809","nonce":21047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6811","nonce":21047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6810","nonce":21047}]},"public_addr":"172.21.15.129:6808/21047","cluster_addr":"172.21.15.129:6809/21047","heartbeat_back_addr":"172.21.15.129:6811/21047","heartbeat_front_addr":"172.21.15.129:6810/21047","state":["exists","up"]},{"osd":5,"uuid":"174b7f7f-23bc-45c9-9245-7c3edc4b6235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6800","nonce":21048}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6801","nonce":21048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6803","nonce":21048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6802","nonce":21048}]},"public_addr":"172.21.15.129:6800/21048","cluster_addr":"172.21.15.129:6801/21048","heartbeat_back_addr":"172.21.15.129:6803/21048","heartbeat_front_addr":"172.21.15.129:6802/21048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.965457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.813648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.196234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.055175+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.022483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.092059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:44:28.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 10 librados: watch_flush enter 2024-09-18T11:44:28.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 10 librados: watch_flush exit 2024-09-18T11:44:28.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 20 client.4356.objecter shutdown clearing up homeless session... 2024-09-18T11:44:28.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 10 client.4356.objecter successfully canceled tick 2024-09-18T11:44:28.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 >> v1:172.21.15.129:6812/20806 conn(0x7fdee0038200 legacy=0x7fdee003a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 10 monclient: shutdown 2024-09-18T11:44:28.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:28.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 >> v1:172.21.15.112:6789/0 conn(0x7fdefc0a1ea0 legacy=0x7fdefc123840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 shutdown_connections 2024-09-18T11:44:28.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 >> 172.21.15.112:0/2866763170 conn(0x7fdefc067da0 msgr2=0x7fdefc09a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:28.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 shutdown_connections 2024-09-18T11:44:28.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 1 -- 172.21.15.112:0/2866763170 wait complete. 2024-09-18T11:44:28.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.734+0000 7fdf02df0700 1 librados: shutdown 2024-09-18T11:44:28.749 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-18T11:44:19.901214+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 17, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-18T11:44:20.029398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-18T11:44:28.750 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-18T11:44:28.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient: get_monmap_and_config 2024-09-18T11:44:28.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient: build_initial_monmap 2024-09-18T11:44:28.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient: monmap: 2024-09-18T11:44:28.857 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:28.857 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.858 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:28.857818+0000 2024-09-18T11:44:28.858 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:28.857818+0000 2024-09-18T11:44:28.858 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:28.858 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.858 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:28.858 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:28.858 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:28.858 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 1 Processor -- start 2024-09-18T11:44:28.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 1 -- start start 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient: init 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient(hunting): picked mon.noname-c con 0x7fa4380a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient(hunting): picked mon.noname-a con 0x7fa4380a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient(hunting): picked mon.noname-b con 0x7fa4381385b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa438097040 con 0x7fa4380a0750 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa438096ad0 con 0x7fa4380a0380 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa438096d60 con 0x7fa4381385b0 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43f12b700 10 monclient(hunting): authenticate will time out at 840.699413s 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43e92a700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa4381385b0 0x7fa43813a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41702/0 (socket says 172.21.15.112:41702) 2024-09-18T11:44:28.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.854+0000 7fa43e92a700 1 -- 172.21.15.112:0/1351594562 learned_addr learned my addr 172.21.15.112:0/1351594562 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43e92a700 1 -- 172.21.15.112:0/1351594562 >> v1:172.21.15.112:6790/0 conn(0x7fa4381385b0 legacy=0x7fa43813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d928700 1 -- 172.21.15.112:0/1351594562 >> v1:172.21.15.112:6789/0 conn(0x7fa4380a0750 legacy=0x7fa4380a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43e129700 1 -- 172.21.15.112:0/1351594562 >> v1:172.21.15.129:6789/0 conn(0x7fa4380a0380 legacy=0x7fa438097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804852506 0 0) 0x7fa438096d60 con 0x7fa4381385b0 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): my global_id is 4379 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa424003570 con 0x7fa4381385b0 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782466687 0 0) 0x7fa438097040 con 0x7fa4380a0750 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): my global_id is 4359 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa438096d60 con 0x7fa4380a0750 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507171733 0 0) 0x7fa438096ad0 con 0x7fa4380a0380 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient(hunting): my global_id is 4366 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa438097040 con 0x7fa4380a0380 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686371095 0 0) 0x7fa424003570 con 0x7fa4381385b0 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 >> v1:172.21.15.129:6789/0 conn(0x7fa4380a0380 legacy=0x7fa438097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 >> v1:172.21.15.112:6789/0 conn(0x7fa4380a0750 legacy=0x7fa4380a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: _finish_hunting 0 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 monclient: found mon.noname-b 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:28.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4380a1d40 con 0x7fa4381385b0 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: _finish_auth 0 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 5 monclient: authenticate success, global_id 4379 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa430001cc0 con 0x7fa4381385b0 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: dump: 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.866 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa430001f50 con 0x7fa4381385b0 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 4 set_mon_vals no callback set 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 1 -- 172.21.15.112:0/1351594562 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa430004e10 con 0x7fa4381385b0 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:28.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43d127700 10 monclient: dump: 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 10 monclient: shutdown 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 1 -- 172.21.15.112:0/1351594562 >> v1:172.21.15.112:6790/0 conn(0x7fa4381385b0 legacy=0x7fa43813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 1 -- 172.21.15.112:0/1351594562 shutdown_connections 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 1 -- 172.21.15.112:0/1351594562 >> 172.21.15.112:0/1351594562 conn(0x7fa438067680 msgr2=0x7fa438067a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 1 -- 172.21.15.112:0/1351594562 shutdown_connections 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 1 -- 172.21.15.112:0/1351594562 wait complete. 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 10 monclient: build_initial_monmap 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.858+0000 7fa43f12b700 10 monclient: monmap: 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:28.869 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:28.870 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 librados: starting msgr at 2024-09-18T11:44:28.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 librados: starting objecter 2024-09-18T11:44:28.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 Processor -- start 2024-09-18T11:44:28.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 -- start start 2024-09-18T11:44:28.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 librados: setting wanted keys 2024-09-18T11:44:28.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 librados: calling monclient init 2024-09-18T11:44:28.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 10 monclient: init 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 10 monclient(hunting): picked mon.noname-a con 0x7fa4380a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 10 monclient(hunting): picked mon.noname-b con 0x7fa4380a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 10 monclient(hunting): picked mon.noname-c con 0x7fa4381385b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4380a2080 con 0x7fa4380a0380 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa43813b510 con 0x7fa4380a0750 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa43811fc30 con 0x7fa4381385b0 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43f12b700 10 monclient(hunting): authenticate will time out at 840.704721s 2024-09-18T11:44:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43e129700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa4380a0380 0x7fa43811bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35422/0 (socket says 172.21.15.112:35422) 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43e92a700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa4381385b0 0x7fa43814a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:41704/0 (socket says 172.21.15.112:41704) 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43e129700 1 -- 172.21.15.112:0/3090893728 learned_addr learned my addr 172.21.15.112:0/3090893728 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43e92a700 1 -- 172.21.15.112:0/3090893728 >> v1:172.21.15.112:6790/0 conn(0x7fa4381385b0 legacy=0x7fa43814a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 client.?.objecter ms_handle_connect 0x7fa4381385b0 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 client.?.objecter ms_handle_connect 0x7fa4380a0380 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43e129700 1 -- 172.21.15.112:0/3090893728 >> v1:172.21.15.112:6789/0 conn(0x7fa4380a0380 legacy=0x7fa43811bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43d928700 1 -- 172.21.15.112:0/3090893728 >> v1:172.21.15.129:6789/0 conn(0x7fa4380a0750 legacy=0x7fa43811c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411760197 0 0) 0x7fa43811fc30 con 0x7fa4381385b0 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): my global_id is 4382 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa41c004f10 con 0x7fa4381385b0 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa43e129700 1 --1- 172.21.15.112:0/3090893728 >> v1:172.21.15.112:6789/0 conn(0x7fa4380a0380 0x7fa43811bff0 :-1 s=OPENED pgs=119 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7fa4380a2080 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 client.?.objecter ms_handle_connect 0x7fa4380a0750 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257860892 0 0) 0x7fa43813b510 con 0x7fa4380a0750 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): my global_id is 4369 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa43811fc30 con 0x7fa4380a0750 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 44622713 0 0) 0x7fa41c004f10 con 0x7fa4381385b0 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa43813b510 con 0x7fa4381385b0 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa430004bb0 con 0x7fa4381385b0 2024-09-18T11:44:28.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.862+0000 7fa42a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1003410376 0 0) 0x7fa43813b510 con 0x7fa4381385b0 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 >> v1:172.21.15.129:6789/0 conn(0x7fa4380a0750 legacy=0x7fa43811c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 >> v1:172.21.15.112:6789/0 conn(0x7fa4380a0380 legacy=0x7fa43811bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 monclient: found mon.c 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43811ce10 con 0x7fa4381385b0 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300787311 0 0) 0x7fa4380a2080 con 0x7fa4380a0380 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa430004df0 con 0x7fa4381385b0 2024-09-18T11:44:28.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa430005560 con 0x7fa4381385b0 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 monclient: dump: 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43c926700 4 set_mon_vals no callback set 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 5 monclient: authenticate success, global_id 4382 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 10 monclient: _renew_subs 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4381375f0 con 0x7fa4381385b0 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 10 monclient: _renew_subs 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43811e010 con 0x7fa4381385b0 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 1 librados: init done 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa43000fe60 con 0x7fa4381385b0 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fa430043b50 con 0x7fa4381385b0 2024-09-18T11:44:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 10 client.4382.objecter ms_dispatch 0x7fa438000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 3 client.4382.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 3 client.4382.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa42a7fc700 20 client.4382.objecter dump_active .. 0 homeless 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43805f2e0 con 0x7fa4381385b0 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.866+0000 7fa43e129700 10 monclient: get_auth_request con 0x7fa41c038160 auth_method 0 2024-09-18T11:44:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.870+0000 7fa42a7fc700 10 client.4382.objecter ms_handle_connect 0x7fa41c038160 2024-09-18T11:44:28.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.874+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fa430014b00 con 0x7fa4381385b0 2024-09-18T11:44:28.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.874+0000 7fa42a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:28.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:28.874+0000 7fa42a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:29.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.038+0000 7fa43f12b700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T11:44:29.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.038+0000 7fa43f12b700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T11:44:29.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.038+0000 7fa43f12b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.038+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa43811f3a0 con 0x7fa4381385b0 2024-09-18T11:44:29.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.042+0000 7fa42a7fc700 1 -- 172.21.15.112:0/3090893728 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (unknown 1614612280 0 265718887) 0x7fa430014920 con 0x7fa4381385b0 2024-09-18T11:44:29.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.042+0000 7fa42a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T11:44:29.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.042+0000 7fa42a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:29.045 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 8 2024-09-18T11:44:29.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 10 librados: watch_flush enter 2024-09-18T11:44:29.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 10 librados: watch_flush exit 2024-09-18T11:44:29.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 20 client.4382.objecter shutdown clearing up homeless session... 2024-09-18T11:44:29.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 10 client.4382.objecter successfully canceled tick 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 >> v1:172.21.15.129:6812/20806 conn(0x7fa41c038160 legacy=0x7fa41c03a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 10 monclient: shutdown 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 >> v1:172.21.15.112:6790/0 conn(0x7fa4381385b0 legacy=0x7fa43814a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 shutdown_connections 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 >> 172.21.15.112:0/3090893728 conn(0x7fa438067680 msgr2=0x7fa438139c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 shutdown_connections 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 1 -- 172.21.15.112:0/3090893728 wait complete. 2024-09-18T11:44:29.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.046+0000 7fa43f12b700 1 librados: shutdown 2024-09-18T11:44:29.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-18T11:44:29.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient: get_monmap_and_config 2024-09-18T11:44:29.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient: build_initial_monmap 2024-09-18T11:44:29.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient: monmap: 2024-09-18T11:44:29.176 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:29.176791+0000 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:29.176791+0000 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:29.177 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 1 Processor -- start 2024-09-18T11:44:29.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 1 -- start start 2024-09-18T11:44:29.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient: init 2024-09-18T11:44:29.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient(hunting): picked mon.noname-a con 0x7f0c2c0955a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient(hunting): picked mon.noname-c con 0x7f0c2c09cfd0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient(hunting): picked mon.noname-b con 0x7f0c2c09d3d0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c063a60 con 0x7f0c2c0955a0 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c09a460 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c099410 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c30e49700 10 monclient(hunting): authenticate will time out at 841.017869s 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c2bfff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0c2c09d3d0 0x7f0c2c137b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36424/0 (socket says 172.21.15.112:36424) 2024-09-18T11:44:29.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.174+0000 7f0c2bfff700 1 -- 172.21.15.112:0/2390991255 learned_addr learned my addr 172.21.15.112:0/2390991255 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:29.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2bfff700 1 -- 172.21.15.112:0/2390991255 >> v1:172.21.15.112:6790/0 conn(0x7f0c2c09d3d0 legacy=0x7f0c2c137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2b7fe700 1 -- 172.21.15.112:0/2390991255 >> v1:172.21.15.112:6789/0 conn(0x7f0c2c0955a0 legacy=0x7f0c2c09b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c23fff700 1 -- 172.21.15.112:0/2390991255 >> v1:172.21.15.129:6789/0 conn(0x7f0c2c09cfd0 legacy=0x7f0c2c097a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528849606 0 0) 0x7f0c2c099410 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): my global_id is 4385 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c18003570 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005423370 0 0) 0x7f0c2c063a60 con 0x7f0c2c0955a0 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): my global_id is 4365 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2c099410 con 0x7f0c2c0955a0 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741281207 0 0) 0x7f0c2c09a460 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient(hunting): my global_id is 4372 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2c063a60 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3505699839 0 0) 0x7f0c18003570 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 >> v1:172.21.15.129:6789/0 conn(0x7f0c2c09cfd0 legacy=0x7f0c2c097a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 >> v1:172.21.15.112:6789/0 conn(0x7f0c2c0955a0 legacy=0x7f0c2c09b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 monclient: found mon.noname-b 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c09b830 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: _finish_auth 0 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c24001d80 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 5 monclient: authenticate success, global_id 4385 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: dump: 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c24001f60 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 1 -- 172.21.15.112:0/2390991255 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c24004e10 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 4 set_mon_vals no callback set 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c2affd700 10 monclient: dump: 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.186 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 10 monclient: shutdown 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 -- 172.21.15.112:0/2390991255 >> v1:172.21.15.112:6790/0 conn(0x7f0c2c09d3d0 legacy=0x7f0c2c137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 -- 172.21.15.112:0/2390991255 shutdown_connections 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 -- 172.21.15.112:0/2390991255 >> 172.21.15.112:0/2390991255 conn(0x7f0c2c0a15d0 msgr2=0x7f0c2c0a19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 -- 172.21.15.112:0/2390991255 shutdown_connections 2024-09-18T11:44:29.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 -- 172.21.15.112:0/2390991255 wait complete. 2024-09-18T11:44:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 10 monclient: build_initial_monmap 2024-09-18T11:44:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 10 monclient: monmap: 2024-09-18T11:44:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:29.192 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 librados: starting msgr at 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 librados: starting objecter 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 Processor -- start 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 -- start start 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 librados: setting wanted keys 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 1 librados: calling monclient init 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.178+0000 7f0c30e49700 10 monclient: init 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 10 monclient(hunting): picked mon.noname-c con 0x7f0c2c0955a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 10 monclient(hunting): picked mon.noname-a con 0x7f0c2c09cfd0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 10 monclient(hunting): picked mon.noname-b con 0x7f0c2c09d3d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c06fc00 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c0972e0 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2c12ad20 con 0x7f0c2c0955a0 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c30e49700 10 monclient(hunting): authenticate will time out at 841.024748s 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c23fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0c2c09cfd0 0x7f0c2c129f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36614/0 (socket says 172.21.15.112:36614) 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c23fff700 1 -- 172.21.15.112:0/2862522445 learned_addr learned my addr 172.21.15.112:0/2862522445 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c23fff700 1 -- 172.21.15.112:0/2862522445 >> v1:172.21.15.112:6789/0 conn(0x7f0c2c09cfd0 legacy=0x7f0c2c129f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 client.?.objecter ms_handle_connect 0x7f0c2c09cfd0 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 client.?.objecter ms_handle_connect 0x7f0c2c0955a0 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c2b7fe700 1 -- 172.21.15.112:0/2862522445 >> v1:172.21.15.112:6790/0 conn(0x7f0c2c0955a0 legacy=0x7f0c2c129880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411586472 0 0) 0x7f0c2c12ad20 con 0x7f0c2c0955a0 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): my global_id is 4388 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c0c004f10 con 0x7f0c2c0955a0 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013134231 0 0) 0x7f0c2c06fc00 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): my global_id is 4368 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2c12ad20 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 client.?.objecter ms_handle_connect 0x7f0c2c09d3d0 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c2bfff700 1 -- 172.21.15.112:0/2862522445 >> v1:172.21.15.129:6789/0 conn(0x7f0c2c09d3d0 legacy=0x7f0c2c147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 292700926 0 0) 0x7f0c0c004f10 con 0x7f0c2c0955a0 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c2c06fc00 con 0x7f0c2c0955a0 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c140019c0 con 0x7f0c2c0955a0 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): dump: 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.196 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280904875 0 0) 0x7f0c2c0972e0 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 10 monclient(hunting): my global_id is 4375 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c0c004f10 con 0x7f0c2c09d3d0 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4281206107 0 0) 0x7f0c2c12ad20 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.182+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c2c0972e0 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c1c001cb0 con 0x7f0c2c09cfd0 2024-09-18T11:44:29.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient(hunting): dump: 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1737932904 0 0) 0x7f0c2c06fc00 con 0x7f0c2c0955a0 2024-09-18T11:44:29.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 >> v1:172.21.15.129:6789/0 conn(0x7f0c2c09d3d0 legacy=0x7f0c2c147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 >> v1:172.21.15.112:6789/0 conn(0x7f0c2c09cfd0 legacy=0x7f0c2c129f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 monclient: found mon.c 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c12c900 con 0x7f0c2c0955a0 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 5 monclient: authenticate success, global_id 4388 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 10 monclient: _renew_subs 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c12b3e0 con 0x7f0c2c0955a0 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:29.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 10 monclient: _renew_subs 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2c12a840 con 0x7f0c2c0955a0 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 1 librados: init done 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c140022b0 con 0x7f0c2c0955a0 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c14002600 con 0x7f0c2c0955a0 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c2a7fc700 4 set_mon_vals no callback set 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 monclient: dump: 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.200 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0c14002760 con 0x7f0c2c0955a0 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c100050b0 con 0x7f0c2c0955a0 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7f0c140436b0 con 0x7f0c2c0955a0 2024-09-18T11:44:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 10 client.4388.objecter ms_dispatch 0x7f0c2c000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:29.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 3 client.4388.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:29.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 3 client.4388.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:29.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c237fe700 20 client.4388.objecter dump_active .. 0 homeless 2024-09-18T11:44:29.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.186+0000 7f0c23fff700 10 monclient: get_auth_request con 0x7f0c0c038530 auth_method 0 2024-09-18T11:44:29.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.190+0000 7f0c237fe700 10 client.4388.objecter ms_handle_connect 0x7f0c0c038530 2024-09-18T11:44:29.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.198+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f0c14002ca0 con 0x7f0c2c0955a0 2024-09-18T11:44:29.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.198+0000 7f0c237fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.198+0000 7f0c237fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:29.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.362+0000 7f0c30e49700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T11:44:29.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.362+0000 7f0c30e49700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T11:44:29.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.362+0000 7f0c30e49700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.362+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c100058e0 con 0x7f0c2c0955a0 2024-09-18T11:44:29.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.362+0000 7f0c237fe700 1 -- 172.21.15.112:0/2862522445 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (unknown 4068545244 0 2170607528) 0x7f0c14014420 con 0x7f0c2c0955a0 2024-09-18T11:44:29.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.362+0000 7f0c237fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T11:44:29.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.362+0000 7f0c237fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:29.368 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2024-09-18T11:44:29.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 10 librados: watch_flush enter 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 10 librados: watch_flush exit 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 20 client.4388.objecter shutdown clearing up homeless session... 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 10 client.4388.objecter successfully canceled tick 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 >> v1:172.21.15.129:6812/20806 conn(0x7f0c0c038530 legacy=0x7f0c0c03a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 10 monclient: shutdown 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 >> v1:172.21.15.112:6790/0 conn(0x7f0c2c0955a0 legacy=0x7f0c2c129880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 shutdown_connections 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 >> 172.21.15.112:0/2862522445 conn(0x7f0c2c0a15d0 msgr2=0x7f0c2c094690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 shutdown_connections 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 1 -- 172.21.15.112:0/2862522445 wait complete. 2024-09-18T11:44:29.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.366+0000 7f0c30e49700 1 librados: shutdown 2024-09-18T11:44:29.386 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-18T11:44:29.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.482+0000 7fcd5759e700 10 monclient: get_monmap_and_config 2024-09-18T11:44:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.482+0000 7fcd5759e700 10 monclient: build_initial_monmap 2024-09-18T11:44:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.482+0000 7fcd5759e700 10 monclient: monmap: 2024-09-18T11:44:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:29.488344+0000 2024-09-18T11:44:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:29.488344+0000 2024-09-18T11:44:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:29.489 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 1 Processor -- start 2024-09-18T11:44:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 1 -- start start 2024-09-18T11:44:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 10 monclient: init 2024-09-18T11:44:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 10 monclient(hunting): picked mon.noname-c con 0x7fcd580a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 10 monclient(hunting): picked mon.noname-a con 0x7fcd58098730 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 10 monclient(hunting): picked mon.noname-b con 0x7fcd58138310 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58097af0 con 0x7fcd58098730 2024-09-18T11:44:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58097780 con 0x7fcd580a2040 2024-09-18T11:44:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5809dea0 con 0x7fcd58138310 2024-09-18T11:44:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5759e700 10 monclient(hunting): authenticate will time out at 841.329142s 2024-09-18T11:44:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd55d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcd58098730 0x7fcd5809d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36620/0 (socket says 172.21.15.112:36620) 2024-09-18T11:44:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd55d9b700 1 -- 172.21.15.112:0/2365073190 learned_addr learned my addr 172.21.15.112:0/2365073190 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:29.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd55d9b700 1 -- 172.21.15.112:0/2365073190 >> v1:172.21.15.112:6789/0 conn(0x7fcd58098730 legacy=0x7fcd5809d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd56d9d700 1 -- 172.21.15.112:0/2365073190 >> v1:172.21.15.112:6790/0 conn(0x7fcd58138310 legacy=0x7fcd5813a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5659c700 1 -- 172.21.15.112:0/2365073190 >> v1:172.21.15.129:6789/0 conn(0x7fcd580a2040 legacy=0x7fcd5809d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697176045 0 0) 0x7fcd58097af0 con 0x7fcd58098730 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.486+0000 7fcd5559a700 10 monclient(hunting): my global_id is 4371 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd40003570 con 0x7fcd58098730 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318684494 0 0) 0x7fcd5809dea0 con 0x7fcd58138310 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient(hunting): my global_id is 4391 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd58097af0 con 0x7fcd58138310 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981986145 0 0) 0x7fcd58097780 con 0x7fcd580a2040 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient(hunting): my global_id is 4378 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd5809dea0 con 0x7fcd580a2040 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3162996738 0 0) 0x7fcd58097af0 con 0x7fcd58138310 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 >> v1:172.21.15.129:6789/0 conn(0x7fcd580a2040 legacy=0x7fcd5809d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 >> v1:172.21.15.112:6789/0 conn(0x7fcd58098730 legacy=0x7fcd5809d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient: _finish_hunting 0 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 monclient: found mon.noname-b 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5809e640 con 0x7fcd58138310 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient: _finish_auth 0 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 5 monclient: authenticate success, global_id 4391 2024-09-18T11:44:29.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd48001970 con 0x7fcd58138310 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient: dump: 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.497 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd56d9d700 1 --1- 172.21.15.112:0/2365073190 >> v1:172.21.15.112:6790/0 conn(0x7fcd58138310 0x7fcd5813a6f0 :-1 s=OPENED pgs=120 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7fcd48001c00 config(0 keys) v1 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd48001c00 con 0x7fcd58138310 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 4 set_mon_vals no callback set 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 10 monclient: shutdown 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 -- 172.21.15.112:0/2365073190 >> v1:172.21.15.112:6790/0 conn(0x7fcd58138310 legacy=0x7fcd5813a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 1 -- 172.21.15.112:0/2365073190 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd480020a0 con 0x7fcd58138310 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5559a700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 -- 172.21.15.112:0/2365073190 shutdown_connections 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 -- 172.21.15.112:0/2365073190 >> 172.21.15.112:0/2365073190 conn(0x7fcd58067da0 msgr2=0x7fcd580a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 -- 172.21.15.112:0/2365073190 shutdown_connections 2024-09-18T11:44:29.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 -- 172.21.15.112:0/2365073190 wait complete. 2024-09-18T11:44:29.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 10 monclient: build_initial_monmap 2024-09-18T11:44:29.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 10 monclient: monmap: 2024-09-18T11:44:29.503 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:29.504 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:29.504 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:29.504 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:29.504 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:29.504 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.504 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:29.721 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 librados: starting msgr at 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 librados: starting objecter 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 Processor -- start 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 -- start start 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 librados: setting wanted keys 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 1 librados: calling monclient init 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.490+0000 7fcd5759e700 10 monclient: init 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 10 monclient(hunting): picked mon.noname-b con 0x7fcd58098730 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 10 monclient(hunting): picked mon.noname-a con 0x7fcd580a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:29.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 10 monclient(hunting): picked mon.noname-c con 0x7fcd58138310 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58096a60 con 0x7fcd580a2040 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5813b140 con 0x7fcd58098730 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58135d80 con 0x7fcd58138310 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5759e700 10 monclient(hunting): authenticate will time out at 841.336653s 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd55d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcd580a2040 0x7fcd58137850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36630/0 (socket says 172.21.15.112:36630) 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd55d9b700 1 -- 172.21.15.112:0/3029466359 learned_addr learned my addr 172.21.15.112:0/3029466359 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd55d9b700 1 -- 172.21.15.112:0/3029466359 >> v1:172.21.15.112:6789/0 conn(0x7fcd580a2040 legacy=0x7fcd58137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 client.?.objecter ms_handle_connect 0x7fcd580a2040 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd56d9d700 1 -- 172.21.15.112:0/3029466359 >> v1:172.21.15.112:6790/0 conn(0x7fcd58138310 legacy=0x7fcd58149870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 client.?.objecter ms_handle_connect 0x7fcd58138310 2024-09-18T11:44:29.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331244550 0 0) 0x7fcd58096a60 con 0x7fcd580a2040 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): my global_id is 4374 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd38004f10 con 0x7fcd580a2040 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487349883 0 0) 0x7fcd58135d80 con 0x7fcd58138310 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): my global_id is 4394 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd58096a60 con 0x7fcd58138310 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 client.?.objecter ms_handle_connect 0x7fcd58098730 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd5659c700 1 -- 172.21.15.112:0/3029466359 >> v1:172.21.15.129:6789/0 conn(0x7fcd58098730 legacy=0x7fcd581414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2763547127 0 0) 0x7fcd58096a60 con 0x7fcd58138310 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd58135d80 con 0x7fcd58138310 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd480018f0 con 0x7fcd58138310 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): dump: 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.725 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411355917 0 0) 0x7fcd5813b140 con 0x7fcd58098730 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): my global_id is 4381 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd58096a60 con 0x7fcd58098730 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3454763639 0 0) 0x7fcd38004f10 con 0x7fcd580a2040 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd5813b140 con 0x7fcd580a2040 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd50001d30 con 0x7fcd580a2040 2024-09-18T11:44:29.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 10 monclient(hunting): dump: 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.494+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1493117304 0 0) 0x7fcd58135d80 con 0x7fcd58138310 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 >> v1:172.21.15.129:6789/0 conn(0x7fcd58098730 legacy=0x7fcd581414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 >> v1:172.21.15.112:6789/0 conn(0x7fcd580a2040 legacy=0x7fcd58137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 1 monclient: found mon.c 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd58141c10 con 0x7fcd58138310 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 5 monclient: authenticate success, global_id 4394 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 10 monclient: _renew_subs 2024-09-18T11:44:29.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd58134b90 con 0x7fcd58138310 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 10 monclient: _renew_subs 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd58135a80 con 0x7fcd58138310 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 1 librados: init done 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd480013b0 con 0x7fcd58138310 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd480027b0 con 0x7fcd58138310 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd54d99700 4 set_mon_vals no callback set 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 10 monclient: dump: 2024-09-18T11:44:29.729 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fcd48002910 con 0x7fcd58138310 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:29.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3c0050b0 con 0x7fcd58138310 2024-09-18T11:44:29.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fcd48043730 con 0x7fcd58138310 2024-09-18T11:44:29.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 10 client.4394.objecter ms_dispatch 0x7fcd58000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:29.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 3 client.4394.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:29.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 3 client.4394.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:29.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd467fc700 20 client.4394.objecter dump_active .. 0 homeless 2024-09-18T11:44:29.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.498+0000 7fcd5659c700 10 monclient: get_auth_request con 0x7fcd380385e0 auth_method 0 2024-09-18T11:44:29.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.502+0000 7fcd467fc700 10 client.4394.objecter ms_handle_connect 0x7fcd380385e0 2024-09-18T11:44:29.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.506+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fcd48014aa0 con 0x7fcd58138310 2024-09-18T11:44:29.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.506+0000 7fcd467fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.506+0000 7fcd467fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:29.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.666+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd3c000b90 con 0x7fcd380385e0 2024-09-18T11:44:29.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd467fc700 1 -- 172.21.15.112:0/3029466359 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41059 (unknown 2965378022 0 931775123) 0x7fcd3c000b90 con 0x7fcd380385e0 2024-09-18T11:44:29.733 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 10 librados: watch_flush enter 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 10 librados: watch_flush exit 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 20 client.4394.objecter shutdown clearing up homeless session... 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 10 client.4394.objecter successfully canceled tick 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 >> v1:172.21.15.129:6812/20806 conn(0x7fcd380385e0 legacy=0x7fcd3803aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 10 monclient: shutdown 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 >> v1:172.21.15.112:6790/0 conn(0x7fcd58138310 legacy=0x7fcd58149870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 shutdown_connections 2024-09-18T11:44:29.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 >> 172.21.15.112:0/3029466359 conn(0x7fcd58067da0 msgr2=0x7fcd5813b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:29.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 shutdown_connections 2024-09-18T11:44:29.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 1 -- 172.21.15.112:0/3029466359 wait complete. 2024-09-18T11:44:29.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.674+0000 7fcd5759e700 1 librados: shutdown 2024-09-18T11:44:29.736 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T11:44:29.736 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-18T11:44:28.214916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1785164,"kb_used_data":972,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564445876,"statfs":{"total":579820584960,"available":577992577024,"internally_reserved":0,"allocated":995328,"data_stored":1329150,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.017797"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.298671+0000","last_change":"2024-09-18T11:44:24.299141+0000","last_active":"2024-09-18T11:44:24.298671+0000","last_peered":"2024-09-18T11:44:24.298671+0000","last_clean":"2024-09-18T11:44:24.298671+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T11:44:24.298671+0000","last_undegraded":"2024-09-18T11:44:24.298671+0000","last_fullsized":"2024-09-18T11:44:24.298671+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:35:52.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.207977+0000","last_change":"2024-09-18T11:44:24.208317+0000","last_active":"2024-09-18T11:44:24.207977+0000","last_peered":"2024-09-18T11:44:24.207977+0000","last_clean":"2024-09-18T11:44:24.207977+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T11:44:24.207977+0000","last_undegraded":"2024-09-18T11:44:24.207977+0000","last_fullsized":"2024-09-18T11:44:24.207977+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:54.979951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.212606+0000","last_change":"2024-09-18T11:44:24.213224+0000","last_active":"2024-09-18T11:44:24.212606+0000","last_peered":"2024-09-18T11:44:24.212606+0000","last_clean":"2024-09-18T11:44:24.212606+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T11:44:24.212606+0000","last_undegraded":"2024-09-18T11:44:24.212606+0000","last_fullsized":"2024-09-18T11:44:24.212606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:47.310939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.507701+0000","last_change":"2024-09-18T11:44:24.507933+0000","last_active":"2024-09-18T11:44:24.507701+0000","last_peered":"2024-09-18T11:44:24.507701+0000","last_clean":"2024-09-18T11:44:24.507701+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T11:44:24.507701+0000","last_undegraded":"2024-09-18T11:44:24.507701+0000","last_fullsized":"2024-09-18T11:44:24.507701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:56.697013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":101,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.069293+0000","last_change":"2024-09-18T11:44:21.097910+0000","last_active":"2024-09-18T11:44:24.069293+0000","last_peered":"2024-09-18T11:44:24.069293+0000","last_clean":"2024-09-18T11:44:24.069293+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T11:44:24.069293+0000","last_undegraded":"2024-09-18T11:44:24.069293+0000","last_fullsized":"2024-09-18T11:44:24.069293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:48.394073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.507643+0000","last_change":"2024-09-18T11:44:24.508079+0000","last_active":"2024-09-18T11:44:24.507643+0000","last_peered":"2024-09-18T11:44:24.507643+0000","last_clean":"2024-09-18T11:44:24.507643+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T11:44:24.507643+0000","last_undegraded":"2024-09-18T11:44:24.507643+0000","last_fullsized":"2024-09-18T11:44:24.507643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:41:30.648862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.214482+0000","last_change":"2024-09-18T11:44:24.214482+0000","last_active":"2024-09-18T11:44:24.214482+0000","last_peered":"2024-09-18T11:44:24.214482+0000","last_clean":"2024-09-18T11:44:24.214482+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T11:44:24.214482+0000","last_undegraded":"2024-09-18T11:44:24.214482+0000","last_fullsized":"2024-09-18T11:44:24.214482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:26:34.055867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.298527+0000","last_change":"2024-09-18T11:44:24.298975+0000","last_active":"2024-09-18T11:44:24.298527+0000","last_peered":"2024-09-18T11:44:24.298527+0000","last_clean":"2024-09-18T11:44:24.298527+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T11:44:24.298527+0000","last_undegraded":"2024-09-18T11:44:24.298527+0000","last_fullsized":"2024-09-18T11:44:24.298527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:27.175495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.207879+0000","last_change":"2024-09-18T11:44:24.208252+0000","last_active":"2024-09-18T11:44:24.207879+0000","last_peered":"2024-09-18T11:44:24.207879+0000","last_clean":"2024-09-18T11:44:24.207879+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T11:44:24.207879+0000","last_undegraded":"2024-09-18T11:44:24.207879+0000","last_fullsized":"2024-09-18T11:44:24.207879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:08:27.257954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":45379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":44380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]}]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":200704,"data_stored":574307,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":45379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":200704,"data_stored":574307,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":151552,"data_stored":45398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:29.737 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:44:29.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient: get_monmap_and_config 2024-09-18T11:44:29.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient: build_initial_monmap 2024-09-18T11:44:29.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient: monmap: 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:29.848622+0000 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:29.848622+0000 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:29.849 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 1 Processor -- start 2024-09-18T11:44:29.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 1 -- start start 2024-09-18T11:44:29.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient: init 2024-09-18T11:44:29.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient(hunting): picked mon.noname-b con 0x7f559c0a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient(hunting): picked mon.noname-a con 0x7f559c0a0e60 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient(hunting): picked mon.noname-c con 0x7f559c097970 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c097550 con 0x7f559c0a0e60 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c096cb0 con 0x7f559c097970 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c0a1440 con 0x7f559c0a0380 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f55a0e80700 10 monclient(hunting): authenticate will time out at 841.689450s 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f559affd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f559c0a0e60 0x7f559c137390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36646/0 (socket says 172.21.15.112:36646) 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f559b7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f559c0a0380 0x7f559c0a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36462/0 (socket says 172.21.15.112:36462) 2024-09-18T11:44:29.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.846+0000 7f559affd700 1 -- 172.21.15.112:0/2045471906 learned_addr learned my addr 172.21.15.112:0/2045471906 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:29.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559b7fe700 1 -- 172.21.15.112:0/2045471906 >> v1:172.21.15.112:6790/0 conn(0x7f559c0a0380 legacy=0x7f559c0a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559affd700 1 -- 172.21.15.112:0/2045471906 >> v1:172.21.15.112:6789/0 conn(0x7f559c0a0e60 legacy=0x7f559c137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559bfff700 1 -- 172.21.15.112:0/2045471906 >> v1:172.21.15.129:6789/0 conn(0x7f559c097970 legacy=0x7f559c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731487324 0 0) 0x7f559c097550 con 0x7f559c0a0e60 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): my global_id is 4377 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5588003570 con 0x7f559c0a0e60 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456433734 0 0) 0x7f559c0a1440 con 0x7f559c0a0380 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): my global_id is 4397 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c097550 con 0x7f559c0a0380 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283597177 0 0) 0x7f559c096cb0 con 0x7f559c097970 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient(hunting): my global_id is 4384 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c0a1440 con 0x7f559c097970 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4148752152 0 0) 0x7f5588003570 con 0x7f559c0a0e60 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 >> v1:172.21.15.112:6790/0 conn(0x7f559c0a0380 legacy=0x7f559c0a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 >> v1:172.21.15.129:6789/0 conn(0x7f559c097970 legacy=0x7f559c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 monclient: found mon.noname-a 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c0a1de0 con 0x7f559c0a0e60 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f558c001cc0 con 0x7f559c0a0e60 2024-09-18T11:44:29.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: dump: 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 5 monclient: authenticate success, global_id 4377 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f558c001f50 con 0x7f559c0a0e60 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 4 set_mon_vals no callback set 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 1 -- 172.21.15.112:0/2045471906 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f558c004e10 con 0x7f559c0a0e60 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f559a7fc700 10 monclient: dump: 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.858 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 10 monclient: shutdown 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 1 -- 172.21.15.112:0/2045471906 >> v1:172.21.15.112:6789/0 conn(0x7f559c0a0e60 legacy=0x7f559c137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 1 -- 172.21.15.112:0/2045471906 shutdown_connections 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 1 -- 172.21.15.112:0/2045471906 >> 172.21.15.112:0/2045471906 conn(0x7f559c068100 msgr2=0x7f559c0a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 1 -- 172.21.15.112:0/2045471906 shutdown_connections 2024-09-18T11:44:29.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 1 -- 172.21.15.112:0/2045471906 wait complete. 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 10 monclient: build_initial_monmap 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.850+0000 7f55a0e80700 10 monclient: monmap: 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.864 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 librados: starting msgr at 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 librados: starting objecter 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 Processor -- start 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 -- start start 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 librados: setting wanted keys 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 librados: calling monclient init 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 10 monclient: init 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 10 monclient(hunting): picked mon.noname-a con 0x7f559c097970 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:29.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 10 monclient(hunting): picked mon.noname-b con 0x7f559c0a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 10 monclient(hunting): picked mon.noname-c con 0x7f559c0a0e60 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c122aa0 con 0x7f559c097970 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c122730 con 0x7f559c0a0380 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c142350 con 0x7f559c0a0e60 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f55a0e80700 10 monclient(hunting): authenticate will time out at 841.697699s 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f559b7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f559c097970 0x7f559c1414c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36650/0 (socket says 172.21.15.112:36650) 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.854+0000 7f559b7fe700 1 -- 172.21.15.112:0/611563538 learned_addr learned my addr 172.21.15.112:0/611563538 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f559bfff700 1 -- 172.21.15.112:0/611563538 >> v1:172.21.15.112:6790/0 conn(0x7f559c0a0e60 legacy=0x7f559c149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 client.?.objecter ms_handle_connect 0x7f559c0a0e60 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 client.?.objecter ms_handle_connect 0x7f559c097970 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f559b7fe700 1 -- 172.21.15.112:0/611563538 >> v1:172.21.15.112:6789/0 conn(0x7f559c097970 legacy=0x7f559c1414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f559affd700 1 -- 172.21.15.112:0/611563538 >> v1:172.21.15.129:6789/0 conn(0x7f559c0a0380 legacy=0x7f559c135330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:29.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225227148 0 0) 0x7f559c142350 con 0x7f559c0a0e60 2024-09-18T11:44:29.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): my global_id is 4400 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c004f10 con 0x7f559c0a0e60 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 client.?.objecter ms_handle_connect 0x7f559c0a0380 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598707228 0 0) 0x7f559c122aa0 con 0x7f559c097970 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): my global_id is 4380 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c142350 con 0x7f559c097970 2024-09-18T11:44:29.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281945265 0 0) 0x7f559c122730 con 0x7f559c0a0380 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): my global_id is 4387 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c122aa0 con 0x7f559c0a0380 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 29229856 0 0) 0x7f559c142350 con 0x7f559c097970 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f559c122730 con 0x7f559c097970 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480989743 0 0) 0x7f557c004f10 con 0x7f559c0a0e60 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f559c142350 con 0x7f559c0a0e60 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5584001ce0 con 0x7f559c097970 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): dump: 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.869 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5594001c00 con 0x7f559c0a0e60 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:29.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient(hunting): dump: 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1192256019 0 0) 0x7f559c122730 con 0x7f559c097970 2024-09-18T11:44:29.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 >> v1:172.21.15.112:6790/0 conn(0x7f559c0a0e60 legacy=0x7f559c149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 >> v1:172.21.15.129:6789/0 conn(0x7f559c0a0380 legacy=0x7f559c135330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 monclient: found mon.a 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c122450 con 0x7f559c097970 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 5 monclient: authenticate success, global_id 4380 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 10 monclient: _renew_subs 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c120920 con 0x7f559c097970 2024-09-18T11:44:29.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:29.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 10 monclient: _renew_subs 2024-09-18T11:44:29.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:29.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c135be0 con 0x7f559c097970 2024-09-18T11:44:29.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 1 librados: init done 2024-09-18T11:44:29.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5584004b60 con 0x7f559c097970 2024-09-18T11:44:29.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:29.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5584004eb0 con 0x7f559c097970 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f5599ffb700 4 set_mon_vals no callback set 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 monclient: dump: 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:29.874 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5584005090 con 0x7f559c097970 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55800050b0 con 0x7f559c097970 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7f55840437e0 con 0x7f559c097970 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 10 client.4380.objecter ms_dispatch 0x7f559c000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 3 client.4380.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 3 client.4380.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f55937fe700 20 client.4380.objecter dump_active .. 0 homeless 2024-09-18T11:44:29.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.858+0000 7f559affd700 10 monclient: get_auth_request con 0x7f557c038400 auth_method 0 2024-09-18T11:44:29.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.862+0000 7f55937fe700 10 client.4380.objecter ms_handle_connect 0x7f557c038400 2024-09-18T11:44:29.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.866+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f55840055d0 con 0x7f559c097970 2024-09-18T11:44:29.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.866+0000 7f55937fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:29.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:29.866+0000 7f55937fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:30.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.026+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5580000b90 con 0x7f557c038400 2024-09-18T11:44:30.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.030+0000 7f55937fe700 1 -- 172.21.15.112:0/611563538 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41059 (unknown 2965378022 0 1966029644) 0x7f5580000b90 con 0x7f557c038400 2024-09-18T11:44:30.033 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 10 librados: watch_flush enter 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 10 librados: watch_flush exit 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 20 client.4380.objecter shutdown clearing up homeless session... 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 10 client.4380.objecter successfully canceled tick 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 >> v1:172.21.15.129:6812/20806 conn(0x7f557c038400 legacy=0x7f557c03a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 10 monclient: shutdown 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 >> v1:172.21.15.112:6789/0 conn(0x7f559c097970 legacy=0x7f559c1414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 shutdown_connections 2024-09-18T11:44:30.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 >> 172.21.15.112:0/611563538 conn(0x7f559c068100 msgr2=0x7f559c09b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 shutdown_connections 2024-09-18T11:44:30.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 1 -- 172.21.15.112:0/611563538 wait complete. 2024-09-18T11:44:30.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.034+0000 7f55a0e80700 1 librados: shutdown 2024-09-18T11:44:30.039 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T11:44:30.051 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-18T11:44:29.959290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1785164,"kb_used_data":972,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564445876,"statfs":{"total":579820584960,"available":577992577024,"internally_reserved":0,"allocated":995328,"data_stored":1329150,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.762170"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.298671+0000","last_change":"2024-09-18T11:44:24.299141+0000","last_active":"2024-09-18T11:44:24.298671+0000","last_peered":"2024-09-18T11:44:24.298671+0000","last_clean":"2024-09-18T11:44:24.298671+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T11:44:24.298671+0000","last_undegraded":"2024-09-18T11:44:24.298671+0000","last_fullsized":"2024-09-18T11:44:24.298671+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:35:52.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.207977+0000","last_change":"2024-09-18T11:44:24.208317+0000","last_active":"2024-09-18T11:44:24.207977+0000","last_peered":"2024-09-18T11:44:24.207977+0000","last_clean":"2024-09-18T11:44:24.207977+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T11:44:24.207977+0000","last_undegraded":"2024-09-18T11:44:24.207977+0000","last_fullsized":"2024-09-18T11:44:24.207977+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:54.979951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.212606+0000","last_change":"2024-09-18T11:44:24.213224+0000","last_active":"2024-09-18T11:44:24.212606+0000","last_peered":"2024-09-18T11:44:24.212606+0000","last_clean":"2024-09-18T11:44:24.212606+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T11:44:24.212606+0000","last_undegraded":"2024-09-18T11:44:24.212606+0000","last_fullsized":"2024-09-18T11:44:24.212606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:47.310939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.507701+0000","last_change":"2024-09-18T11:44:24.507933+0000","last_active":"2024-09-18T11:44:24.507701+0000","last_peered":"2024-09-18T11:44:24.507701+0000","last_clean":"2024-09-18T11:44:24.507701+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T11:44:24.507701+0000","last_undegraded":"2024-09-18T11:44:24.507701+0000","last_fullsized":"2024-09-18T11:44:24.507701+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:56.697013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":101,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.069293+0000","last_change":"2024-09-18T11:44:21.097910+0000","last_active":"2024-09-18T11:44:24.069293+0000","last_peered":"2024-09-18T11:44:24.069293+0000","last_clean":"2024-09-18T11:44:24.069293+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T11:44:24.069293+0000","last_undegraded":"2024-09-18T11:44:24.069293+0000","last_fullsized":"2024-09-18T11:44:24.069293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:48.394073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.507643+0000","last_change":"2024-09-18T11:44:24.508079+0000","last_active":"2024-09-18T11:44:24.507643+0000","last_peered":"2024-09-18T11:44:24.507643+0000","last_clean":"2024-09-18T11:44:24.507643+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T11:44:24.507643+0000","last_undegraded":"2024-09-18T11:44:24.507643+0000","last_fullsized":"2024-09-18T11:44:24.507643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:41:30.648862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.214482+0000","last_change":"2024-09-18T11:44:24.214482+0000","last_active":"2024-09-18T11:44:24.214482+0000","last_peered":"2024-09-18T11:44:24.214482+0000","last_clean":"2024-09-18T11:44:24.214482+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T11:44:24.214482+0000","last_undegraded":"2024-09-18T11:44:24.214482+0000","last_fullsized":"2024-09-18T11:44:24.214482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:26:34.055867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.298527+0000","last_change":"2024-09-18T11:44:24.298975+0000","last_active":"2024-09-18T11:44:24.298527+0000","last_peered":"2024-09-18T11:44:24.298527+0000","last_clean":"2024-09-18T11:44:24.298527+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T11:44:24.298527+0000","last_undegraded":"2024-09-18T11:44:24.298527+0000","last_fullsized":"2024-09-18T11:44:24.298527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:27.175495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:44:24.207879+0000","last_change":"2024-09-18T11:44:24.208252+0000","last_active":"2024-09-18T11:44:24.207879+0000","last_peered":"2024-09-18T11:44:24.207879+0000","last_clean":"2024-09-18T11:44:24.207879+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T11:44:24.207879+0000","last_undegraded":"2024-09-18T11:44:24.207879+0000","last_fullsized":"2024-09-18T11:44:24.207879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:08:27.257954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":45379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":44380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]}]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":200704,"data_stored":574307,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":147456,"data_stored":45379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":200704,"data_stored":574307,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":151552,"data_stored":45398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:44:30.052 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-18T11:44:30.052 DEBUG:tasks.mon_thrash:Yielding 2024-09-18T11:44:30.052 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T11:44:30.069 INFO:tasks.workunit:Pulling workunits from ref 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:44:30.069 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T11:44:30.069 DEBUG:teuthology.orchestra.run.smithi129:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-18T11:44:30.070 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-18T11:44:30.070 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726659870, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-18T11:44:30.070 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:44:30.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:44:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:44:30.080 INFO:teuthology.orchestra.run.smithi129.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-18T11:44:30.081 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T11:44:30.128 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-18T11:44:30.128 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-18T11:44:30.135 INFO:tasks.workunit:timeout=3h 2024-09-18T11:44:30.135 INFO:tasks.workunit:cleanup=True 2024-09-18T11:44:30.135 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:44:30.143 INFO:tasks.workunit.client.0.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-18T11:44:30.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.166+0000 7fe598e28700 10 monclient: get_monmap_and_config 2024-09-18T11:44:30.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.166+0000 7fe598e28700 10 monclient: build_initial_monmap 2024-09-18T11:44:30.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.166+0000 7fe598e28700 10 monclient: monmap: 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:30.171763+0000 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:30.171763+0000 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:30.172 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 1 Processor -- start 2024-09-18T11:44:30.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 1 -- start start 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 10 monclient: init 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 10 monclient(hunting): picked mon.noname-c con 0x7fe5940a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 10 monclient(hunting): picked mon.noname-a con 0x7fe5940a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 10 monclient(hunting): picked mon.noname-b con 0x7fe5941385b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe594097270 con 0x7fe5940a0750 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940a3170 con 0x7fe5940a0380 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe594096cb0 con 0x7fe5941385b0 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe598e28700 10 monclient(hunting): authenticate will time out at 842.013034s 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe58affd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe5940a0750 0x7fe5940a0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36662/0 (socket says 172.21.15.112:36662) 2024-09-18T11:44:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe58affd700 1 -- 172.21.15.112:0/3937938913 learned_addr learned my addr 172.21.15.112:0/3937938913 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe593fff700 1 --1- 172.21.15.112:0/3937938913 >> v1:172.21.15.112:6790/0 conn(0x7fe5941385b0 0x7fe59413a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36488/0 (socket says 172.21.15.112:36488) 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe58affd700 1 -- 172.21.15.112:0/3937938913 >> v1:172.21.15.112:6789/0 conn(0x7fe5940a0750 legacy=0x7fe5940a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe5937fe700 1 -- 172.21.15.112:0/3937938913 >> v1:172.21.15.129:6789/0 conn(0x7fe5940a0380 legacy=0x7fe594097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe593fff700 1 -- 172.21.15.112:0/3937938913 >> v1:172.21.15.112:6790/0 conn(0x7fe5941385b0 legacy=0x7fe59413a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.170+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205572429 0 0) 0x7fe594097270 con 0x7fe5940a0750 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): my global_id is 4383 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe580003570 con 0x7fe5940a0750 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431527282 0 0) 0x7fe594096cb0 con 0x7fe5941385b0 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): my global_id is 4403 2024-09-18T11:44:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594097270 con 0x7fe5941385b0 2024-09-18T11:44:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675084998 0 0) 0x7fe5940a3170 con 0x7fe5940a0380 2024-09-18T11:44:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient(hunting): my global_id is 4390 2024-09-18T11:44:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594096cb0 con 0x7fe5940a0380 2024-09-18T11:44:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2056460057 0 0) 0x7fe594097270 con 0x7fe5941385b0 2024-09-18T11:44:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 >> v1:172.21.15.129:6789/0 conn(0x7fe5940a0380 legacy=0x7fe594097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 >> v1:172.21.15.112:6789/0 conn(0x7fe5940a0750 legacy=0x7fe5940a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 monclient: found mon.noname-b 2024-09-18T11:44:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940a1420 con 0x7fe5941385b0 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 5 monclient: authenticate success, global_id 4403 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe58c001ca0 con 0x7fe5941385b0 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: dump: 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe58c001f30 con 0x7fe5941385b0 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 1 -- 172.21.15.112:0/3937938913 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe58c002120 con 0x7fe5941385b0 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 4 set_mon_vals no callback set 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe592ffd700 10 monclient: dump: 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient: shutdown 2024-09-18T11:44:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- 172.21.15.112:0/3937938913 >> v1:172.21.15.112:6790/0 conn(0x7fe5941385b0 legacy=0x7fe59413a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- 172.21.15.112:0/3937938913 shutdown_connections 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- 172.21.15.112:0/3937938913 >> 172.21.15.112:0/3937938913 conn(0x7fe5940a5990 msgr2=0x7fe5940a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- 172.21.15.112:0/3937938913 shutdown_connections 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- 172.21.15.112:0/3937938913 wait complete. 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient: build_initial_monmap 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient: monmap: 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 librados: starting msgr at 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 librados: starting objecter 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 Processor -- start 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- start start 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 librados: setting wanted keys 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 librados: calling monclient init 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient: init 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient(hunting): picked mon.noname-c con 0x7fe5940a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient(hunting): picked mon.noname-b con 0x7fe5940a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 10 monclient(hunting): picked mon.noname-a con 0x7fe5941385b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe594141b80 con 0x7fe5941385b0 2024-09-18T11:44:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe594098290 con 0x7fe5940a0750 2024-09-18T11:44:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.174+0000 7fe598e28700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59409ef40 con 0x7fe5940a0380 2024-09-18T11:44:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 10 monclient(hunting): authenticate will time out at 842.020043s 2024-09-18T11:44:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe5937fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe5940a0380 0x7fe594141470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36502/0 (socket says 172.21.15.112:36502) 2024-09-18T11:44:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe5937fe700 1 -- 172.21.15.112:0/159901755 learned_addr learned my addr 172.21.15.112:0/159901755 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe5937fe700 1 -- 172.21.15.112:0/159901755 >> v1:172.21.15.112:6790/0 conn(0x7fe5940a0380 legacy=0x7fe594141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 client.?.objecter ms_handle_connect 0x7fe5940a0380 2024-09-18T11:44:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 client.?.objecter ms_handle_connect 0x7fe5941385b0 2024-09-18T11:44:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe593fff700 1 -- 172.21.15.112:0/159901755 >> v1:172.21.15.112:6789/0 conn(0x7fe5941385b0 legacy=0x7fe5941497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 client.?.objecter ms_handle_connect 0x7fe5940a0750 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58affd700 1 -- 172.21.15.112:0/159901755 >> v1:172.21.15.129:6789/0 conn(0x7fe5940a0750 legacy=0x7fe59409ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845615771 0 0) 0x7fe59409ef40 con 0x7fe5940a0380 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): my global_id is 4406 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe574004f10 con 0x7fe5940a0380 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605531979 0 0) 0x7fe594141b80 con 0x7fe5941385b0 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): my global_id is 4386 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe59409ef40 con 0x7fe5941385b0 2024-09-18T11:44:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819159065 0 0) 0x7fe594098290 con 0x7fe5940a0750 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): my global_id is 4393 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594141b80 con 0x7fe5940a0750 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 450849309 0 0) 0x7fe574004f10 con 0x7fe5940a0380 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe594098290 con 0x7fe5940a0380 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe584001c50 con 0x7fe5940a0380 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): dump: 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 728565972 0 0) 0x7fe594098290 con 0x7fe5940a0380 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 >> v1:172.21.15.129:6789/0 conn(0x7fe5940a0750 legacy=0x7fe59409ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 >> v1:172.21.15.112:6789/0 conn(0x7fe5941385b0 legacy=0x7fe5941497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 monclient: found mon.c 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594137cd0 con 0x7fe5940a0380 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient: _finish_auth 0 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 5 monclient: authenticate success, global_id 4406 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 10 monclient: _renew_subs 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe594097fb0 con 0x7fe5940a0380 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 10 monclient: _renew_subs 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe594142230 con 0x7fe5940a0380 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe598e28700 1 librados: init done 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5840022b0 con 0x7fe5940a0380 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe584002640 con 0x7fe5940a0380 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe5927fc700 4 set_mon_vals no callback set 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.178+0000 7fe58bfff700 10 monclient: dump: 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe5840027a0 con 0x7fe5940a0380 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fe5840437b0 con 0x7fe5940a0380 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe58bfff700 10 client.4406.objecter ms_dispatch 0x7fe594000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe58bfff700 3 client.4406.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe58bfff700 3 client.4406.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe598e28700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe598e28700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe598e28700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:30.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe58bfff700 20 client.4406.objecter dump_active .. 0 homeless 2024-09-18T11:44:30.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5780050b0 con 0x7fe5940a0380 2024-09-18T11:44:30.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.182+0000 7fe58affd700 10 monclient: get_auth_request con 0x7fe574038170 auth_method 0 2024-09-18T11:44:30.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.186+0000 7fe58bfff700 10 client.4406.objecter ms_handle_connect 0x7fe574038170 2024-09-18T11:44:30.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.190+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fe584014b40 con 0x7fe5940a0380 2024-09-18T11:44:30.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.190+0000 7fe58bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.190+0000 7fe58bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:30.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.414+0000 7fe598e28700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:44:30.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.414+0000 7fe598e28700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:44:30.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.414+0000 7fe598e28700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:30.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.414+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe5780058e0 con 0x7fe5940a0380 2024-09-18T11:44:30.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe58bfff700 1 -- 172.21.15.112:0/159901755 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2970459725) 0x7fe584014540 con 0x7fe5940a0380 2024-09-18T11:44:30.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe58bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:44:30.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe58bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:30.421 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:44:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 10 librados: watch_flush enter 2024-09-18T11:44:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 10 librados: watch_flush exit 2024-09-18T11:44:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 20 client.4406.objecter shutdown clearing up homeless session... 2024-09-18T11:44:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 10 client.4406.objecter successfully canceled tick 2024-09-18T11:44:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 >> v1:172.21.15.129:6812/20806 conn(0x7fe574038170 legacy=0x7fe57403a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 10 monclient: shutdown 2024-09-18T11:44:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 >> v1:172.21.15.112:6790/0 conn(0x7fe5940a0380 legacy=0x7fe594141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 shutdown_connections 2024-09-18T11:44:30.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 >> 172.21.15.112:0/159901755 conn(0x7fe5940a5990 msgr2=0x7fe594131340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 shutdown_connections 2024-09-18T11:44:30.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 1 -- 172.21.15.112:0/159901755 wait complete. 2024-09-18T11:44:30.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.418+0000 7fe598e28700 1 librados: shutdown 2024-09-18T11:44:30.434 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:44:30.434 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T11:44:30.434 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:44:30.434 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:44:30.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient: get_monmap_and_config 2024-09-18T11:44:30.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient: build_initial_monmap 2024-09-18T11:44:30.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient: monmap: 2024-09-18T11:44:30.545 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:30.545 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.546 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:30.545371+0000 2024-09-18T11:44:30.546 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:30.545371+0000 2024-09-18T11:44:30.546 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:30.546 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.546 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:30.547 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:30.547 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:30.547 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 1 Processor -- start 2024-09-18T11:44:30.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 1 -- start start 2024-09-18T11:44:30.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient: init 2024-09-18T11:44:30.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:30.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:30.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f579c0a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:30.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f579c06c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:30.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f579c06c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:30.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579c06e950 con 0x7f579c06c280 2024-09-18T11:44:30.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579c0a2270 con 0x7f579c06c670 2024-09-18T11:44:30.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579c06d130 con 0x7f579c0a1ea0 2024-09-18T11:44:30.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:30.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579bfff700 10 monclient(hunting): authenticate will time out at 842.386519s 2024-09-18T11:44:30.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579a7fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f579c06c280 0x7f579c135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36690/0 (socket says 172.21.15.112:36690) 2024-09-18T11:44:30.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579affd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f579c0a1ea0 0x7f579c06bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36514/0 (socket says 172.21.15.112:36514) 2024-09-18T11:44:30.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.542+0000 7f579a7fc700 1 -- 172.21.15.112:0/1405152056 learned_addr learned my addr 172.21.15.112:0/1405152056 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:30.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579a7fc700 1 -- 172.21.15.112:0/1405152056 >> v1:172.21.15.112:6789/0 conn(0x7f579c06c280 legacy=0x7f579c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579affd700 1 -- 172.21.15.112:0/1405152056 >> v1:172.21.15.112:6790/0 conn(0x7f579c0a1ea0 legacy=0x7f579c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579b7fe700 1 -- 172.21.15.112:0/1405152056 >> v1:172.21.15.129:6789/0 conn(0x7f579c06c670 legacy=0x7f579c1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325163890 0 0) 0x7f579c06e950 con 0x7f579c06c280 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): my global_id is 4389 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5788003570 con 0x7f579c06c280 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342369867 0 0) 0x7f579c06d130 con 0x7f579c0a1ea0 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): my global_id is 4409 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c06e950 con 0x7f579c0a1ea0 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692710969 0 0) 0x7f579c0a2270 con 0x7f579c06c670 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient(hunting): my global_id is 4396 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c06d130 con 0x7f579c06c670 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716765863 0 0) 0x7f5788003570 con 0x7f579c06c280 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 >> v1:172.21.15.112:6790/0 conn(0x7f579c0a1ea0 legacy=0x7f579c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 >> v1:172.21.15.129:6789/0 conn(0x7f579c06c670 legacy=0x7f579c1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 monclient: found mon.noname-a 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c09e370 con 0x7f579c06c280 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:30.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:30.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 5 monclient: authenticate success, global_id 4389 2024-09-18T11:44:30.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:30.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f578c001940 con 0x7f579c06c280 2024-09-18T11:44:30.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:30.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: dump: 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:30.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f578c001bd0 con 0x7f579c06c280 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 1 -- 172.21.15.112:0/1405152056 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f578c004a90 con 0x7f579c06c280 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 4 set_mon_vals no callback set 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f5799ffb700 10 monclient: dump: 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.558 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 10 monclient: shutdown 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 1 -- 172.21.15.112:0/1405152056 >> v1:172.21.15.112:6789/0 conn(0x7f579c06c280 legacy=0x7f579c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 1 -- 172.21.15.112:0/1405152056 shutdown_connections 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 1 -- 172.21.15.112:0/1405152056 >> 172.21.15.112:0/1405152056 conn(0x7f579c067da0 msgr2=0x7f579c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 1 -- 172.21.15.112:0/1405152056 shutdown_connections 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 1 -- 172.21.15.112:0/1405152056 wait complete. 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 10 monclient: build_initial_monmap 2024-09-18T11:44:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.546+0000 7f579bfff700 10 monclient: monmap: 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 librados: starting msgr at 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 librados: starting objecter 2024-09-18T11:44:30.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 Processor -- start 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 -- start start 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 librados: setting wanted keys 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 librados: calling monclient init 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 10 monclient: init 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f579c06c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f579c06c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f579c0a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579c11e340 con 0x7f579c06c280 2024-09-18T11:44:30.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579c120920 con 0x7f579c06c670 2024-09-18T11:44:30.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579c120640 con 0x7f579c0a1ea0 2024-09-18T11:44:30.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:30.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579bfff700 10 monclient(hunting): authenticate will time out at 842.393402s 2024-09-18T11:44:30.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579b7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f579c0a1ea0 0x7f579c1476c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36516/0 (socket says 172.21.15.112:36516) 2024-09-18T11:44:30.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579b7fe700 1 -- 172.21.15.112:0/968171852 learned_addr learned my addr 172.21.15.112:0/968171852 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579affd700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6789/0 conn(0x7f579c06c280 legacy=0x7f579c13f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 10 client.?.objecter ms_handle_connect 0x7f579c06c280 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 10 client.?.objecter ms_handle_connect 0x7f579c0a1ea0 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f579b7fe700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6790/0 conn(0x7f579c0a1ea0 legacy=0x7f579c1476c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037201935 0 0) 0x7f579c120640 con 0x7f579c0a1ea0 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 10 monclient(hunting): my global_id is 4412 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.550+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f577c004f10 con 0x7f579c0a1ea0 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150052176 0 0) 0x7f579c11e340 con 0x7f579c06c280 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): my global_id is 4392 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c120640 con 0x7f579c06c280 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 client.?.objecter ms_handle_connect 0x7f579c06c670 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579a7fc700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.129:6789/0 conn(0x7f579c06c670 legacy=0x7f579c132230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895426577 0 0) 0x7f579c120920 con 0x7f579c06c670 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): my global_id is 4399 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c11e340 con 0x7f579c06c670 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988861646 0 0) 0x7f579c120640 con 0x7f579c06c280 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f579c120920 con 0x7f579c06c280 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5784004830 con 0x7f579c06c280 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:30.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): dump: 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3064377274 0 0) 0x7f577c004f10 con 0x7f579c0a1ea0 2024-09-18T11:44:30.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f579c120640 con 0x7f579c0a1ea0 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5794002090 con 0x7f579c0a1ea0 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient(hunting): dump: 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.578 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3034424423 0 0) 0x7f579c120920 con 0x7f579c06c280 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6790/0 conn(0x7f579c0a1ea0 legacy=0x7f579c1476c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.129:6789/0 conn(0x7f579c06c670 legacy=0x7f579c132230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 monclient: found mon.a 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c11f400 con 0x7f579c06c280 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 5 monclient: authenticate success, global_id 4392 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 10 monclient: _renew_subs 2024-09-18T11:44:30.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579c13fa60 con 0x7f579c06c280 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 10 monclient: _renew_subs 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f579c132ae0 con 0x7f579c06c280 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 1 librados: init done 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5784004c40 con 0x7f579c06c280 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5784004f90 con 0x7f579c06c280 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:30.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 monclient: dump: 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f57997fa700 4 set_mon_vals no callback set 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f578400f890 con 0x7f579c06c280 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5780006590 con 0x7f5780002840 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7f5784043870 con 0x7f579c06c280 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 10 client.4392.objecter ms_dispatch 0x7f579c000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 3 client.4392.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 3 client.4392.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f5792ffd700 20 client.4392.objecter dump_active .. 0 homeless 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.554+0000 7f579b7fe700 10 monclient: get_auth_request con 0x7f577c038110 auth_method 0 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 client.4392.objecter ms_handle_connect 0x7f5780002840 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 client.4392.objecter resend_mon_ops 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f579a7fc700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6789/0 conn(0x7f5780002840 legacy=0x7f5780004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159615621 0 0) 0x7f5780006590 con 0x7f5780002840 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 monclient(hunting): my global_id is 4395 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c120920 con 0x7f5780002840 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2999874034 0 0) 0x7f579c120920 con 0x7f5780002840 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5780006590 con 0x7f5780002840 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f578c004e30 con 0x7f5780002840 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 monclient: dump: 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.583 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2486659459 0 0) 0x7f5780006590 con 0x7f5780002840 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f578000a140 con 0x7f5780002840 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 client.4392.objecter ms_handle_connect 0x7f577c038110 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f578c006fc0 con 0x7f5780002840 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6789/0 conn(0x7f5780002840 legacy=0x7f5780004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.558+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6789/0 conn(0x7f5780002840 legacy=0x7f5780004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:30.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f579bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:44:30.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5780009670 con 0x7f57800090e0 2024-09-18T11:44:30.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f5792ffd700 10 client.4392.objecter ms_handle_connect 0x7f57800090e0 2024-09-18T11:44:30.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f5792ffd700 10 client.4392.objecter resend_mon_ops 2024-09-18T11:44:30.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f579a7fc700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6789/0 conn(0x7f57800090e0 legacy=0x7f5780008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24297472 0 0) 0x7f5780009670 con 0x7f57800090e0 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f5792ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f5792ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f5792ffd700 10 monclient(hunting): my global_id is 4398 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.570+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5780006590 con 0x7f57800090e0 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2355356174 0 0) 0x7f5780006590 con 0x7f57800090e0 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5780009670 con 0x7f57800090e0 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f578c005490 con 0x7f57800090e0 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 10 monclient: dump: 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 776487735 0 0) 0x7f5780009670 con 0x7f57800090e0 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5780009930 con 0x7f57800090e0 2024-09-18T11:44:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4003 (unknown 0 0 3887131280) 0x7f578c006760 con 0x7f57800090e0 2024-09-18T11:44:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:44:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6789/0 conn(0x7f57800090e0 legacy=0x7f5780008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f5792ffd700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6789/0 conn(0x7f57800090e0 legacy=0x7f5780008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:30.589 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:44:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:44:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:44:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 8, 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 21, 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:44:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:44:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:44:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:44:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:44:30.601 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:44:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 10 librados: watch_flush enter 2024-09-18T11:44:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 10 librados: watch_flush exit 2024-09-18T11:44:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 20 client.4392.objecter shutdown clearing up homeless session... 2024-09-18T11:44:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 10 client.4392.objecter successfully canceled tick 2024-09-18T11:44:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.129:6812/20806 conn(0x7f577c038110 legacy=0x7f577c03a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 10 monclient: shutdown 2024-09-18T11:44:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 >> v1:172.21.15.112:6789/0 conn(0x7f579c06c280 legacy=0x7f579c13f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579b7fe700 1 -- 172.21.15.112:0/968171852 reap_dead start 2024-09-18T11:44:30.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 shutdown_connections 2024-09-18T11:44:30.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 >> 172.21.15.112:0/968171852 conn(0x7f579c067da0 msgr2=0x7f579c09fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 shutdown_connections 2024-09-18T11:44:30.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 1 -- 172.21.15.112:0/968171852 wait complete. 2024-09-18T11:44:30.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.574+0000 7f579bfff700 1 librados: shutdown 2024-09-18T11:44:30.605 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:44:30.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:44:30.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:44:30.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.714+0000 7f75c1979700 10 monclient: get_monmap_and_config 2024-09-18T11:44:30.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.714+0000 7f75c1979700 10 monclient: build_initial_monmap 2024-09-18T11:44:30.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.714+0000 7f75c1979700 10 monclient: monmap: 2024-09-18T11:44:30.719 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:30.719 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.720 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:30.719654+0000 2024-09-18T11:44:30.720 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:30.719654+0000 2024-09-18T11:44:30.720 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:30.720 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.720 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:30.720 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:30.720 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:30.720 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 1 Processor -- start 2024-09-18T11:44:30.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 1 -- start start 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 10 monclient: init 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0a0570 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 10 monclient(hunting): picked mon.noname-c con 0x7f75bc0a1050 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 10 monclient(hunting): picked mon.noname-b con 0x7f75bc097a50 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc097070 con 0x7f75bc0a0570 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc096d90 con 0x7f75bc0a1050 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc097e50 con 0x7f75bc097a50 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75c1979700 10 monclient(hunting): authenticate will time out at 842.561083s 2024-09-18T11:44:30.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75baffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f75bc0a0570 0x7f75bc0a0940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36724/0 (socket says 172.21.15.112:36724) 2024-09-18T11:44:30.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75baffd700 1 -- 172.21.15.112:0/1513605596 learned_addr learned my addr 172.21.15.112:0/1513605596 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:30.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75baffd700 1 -- 172.21.15.112:0/1513605596 >> v1:172.21.15.112:6789/0 conn(0x7f75bc0a0570 legacy=0x7f75bc0a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75bb7fe700 1 -- 172.21.15.112:0/1513605596 >> v1:172.21.15.112:6790/0 conn(0x7f75bc097a50 legacy=0x7f75bc13aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75b3fff700 1 -- 172.21.15.112:0/1513605596 >> v1:172.21.15.129:6789/0 conn(0x7f75bc0a1050 legacy=0x7f75bc137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257858613 0 0) 0x7f75bc097e50 con 0x7f75bc097a50 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 10 monclient(hunting): my global_id is 4415 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a8003570 con 0x7f75bc097a50 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855560442 0 0) 0x7f75bc097070 con 0x7f75bc0a0570 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 10 monclient(hunting): my global_id is 4401 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.718+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc097e50 con 0x7f75bc0a0570 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973310390 0 0) 0x7f75bc096d90 con 0x7f75bc0a1050 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient(hunting): my global_id is 4402 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc097070 con 0x7f75bc0a1050 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2819847029 0 0) 0x7f75bc097e50 con 0x7f75bc0a0570 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 >> v1:172.21.15.112:6790/0 conn(0x7f75bc097a50 legacy=0x7f75bc13aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 >> v1:172.21.15.129:6789/0 conn(0x7f75bc0a1050 legacy=0x7f75bc137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 monclient: found mon.noname-a 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0a2050 con 0x7f75bc0a0570 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:30.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75ac0019e0 con 0x7f75bc0a0570 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 5 monclient: authenticate success, global_id 4401 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: dump: 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.729 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75ac001b40 con 0x7f75bc0a0570 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 1 -- 172.21.15.112:0/1513605596 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75ac0048f0 con 0x7f75bc0a0570 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 4 set_mon_vals no callback set 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75ba7fc700 10 monclient: dump: 2024-09-18T11:44:30.730 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient: shutdown 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- 172.21.15.112:0/1513605596 >> v1:172.21.15.112:6789/0 conn(0x7f75bc0a0570 legacy=0x7f75bc0a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- 172.21.15.112:0/1513605596 shutdown_connections 2024-09-18T11:44:30.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- 172.21.15.112:0/1513605596 >> 172.21.15.112:0/1513605596 conn(0x7f75bc067680 msgr2=0x7f75bc067a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- 172.21.15.112:0/1513605596 shutdown_connections 2024-09-18T11:44:30.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- 172.21.15.112:0/1513605596 wait complete. 2024-09-18T11:44:30.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient: build_initial_monmap 2024-09-18T11:44:30.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient: monmap: 2024-09-18T11:44:30.732 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:30.732 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 librados: starting msgr at 2024-09-18T11:44:30.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 librados: starting objecter 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 Processor -- start 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- start start 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 librados: setting wanted keys 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 librados: calling monclient init 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient: init 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient(hunting): picked mon.noname-c con 0x7f75bc097a50 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0a0570 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient(hunting): picked mon.noname-b con 0x7f75bc0a1050 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc122850 con 0x7f75bc0a0570 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc122570 con 0x7f75bc0a1050 2024-09-18T11:44:30.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc142430 con 0x7f75bc097a50 2024-09-18T11:44:30.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:30.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75c1979700 10 monclient(hunting): authenticate will time out at 842.566450s 2024-09-18T11:44:30.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75baffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f75bc097a50 0x7f75bc134440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36522/0 (socket says 172.21.15.112:36522) 2024-09-18T11:44:30.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.722+0000 7f75baffd700 1 -- 172.21.15.112:0/1616084626 learned_addr learned my addr 172.21.15.112:0/1616084626 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:30.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b3fff700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6789/0 conn(0x7f75bc0a0570 legacy=0x7f75bc146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 client.?.objecter ms_handle_connect 0x7f75bc0a0570 2024-09-18T11:44:30.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 client.?.objecter ms_handle_connect 0x7f75bc097a50 2024-09-18T11:44:30.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75baffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6790/0 conn(0x7f75bc097a50 legacy=0x7f75bc134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75bb7fe700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.129:6789/0 conn(0x7f75bc0a1050 legacy=0x7f75bc14a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 client.?.objecter ms_handle_connect 0x7f75bc0a1050 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780706489 0 0) 0x7f75bc122850 con 0x7f75bc0a0570 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): my global_id is 4404 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c004f10 con 0x7f75bc0a0570 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786687205 0 0) 0x7f75bc142430 con 0x7f75bc097a50 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): my global_id is 4418 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc122850 con 0x7f75bc097a50 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284699144 0 0) 0x7f75bc122570 con 0x7f75bc0a1050 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): my global_id is 4405 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc142430 con 0x7f75bc0a1050 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2541893149 0 0) 0x7f759c004f10 con 0x7f75bc0a0570 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75bc122570 con 0x7f75bc0a0570 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75a4001cc0 con 0x7f75bc0a0570 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): dump: 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.744 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.745 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.745 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.745 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.745 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.745 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:30.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3692559087 0 0) 0x7f75bc122850 con 0x7f75bc097a50 2024-09-18T11:44:30.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f759c004f10 con 0x7f75bc097a50 2024-09-18T11:44:30.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75ac004830 con 0x7f75bc097a50 2024-09-18T11:44:30.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:30.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient(hunting): dump: 2024-09-18T11:44:30.748 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.748 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.748 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.748 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.748 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.748 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2064391343 0 0) 0x7f75bc122570 con 0x7f75bc0a0570 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6790/0 conn(0x7f75bc097a50 legacy=0x7f75bc134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.129:6789/0 conn(0x7f75bc0a1050 legacy=0x7f75bc14a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 monclient: found mon.a 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc121680 con 0x7f75bc0a0570 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 5 monclient: authenticate success, global_id 4404 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 10 monclient: _renew_subs 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc134b50 con 0x7f75bc0a0570 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 10 monclient: _renew_subs 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc120160 con 0x7f75bc0a0570 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 1 librados: init done 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75a40022d0 con 0x7f75bc0a0570 2024-09-18T11:44:30.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75a40025e0 con 0x7f75bc0a0570 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b9ffb700 4 set_mon_vals no callback set 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75b2ffd700 10 monclient: dump: 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.726+0000 7f75c1979700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f75a0006590 con 0x7f75a0002840 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f75a40027f0 con 0x7f75bc0a0570 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7f75a4043710 con 0x7f75bc0a0570 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 client.4404.objecter ms_dispatch 0x7f75bc000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 3 client.4404.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 3 client.4404.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 20 client.4404.objecter dump_active .. 0 homeless 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75bb7fe700 10 monclient: get_auth_request con 0x7f759c0384b0 auth_method 0 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75baffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6790/0 conn(0x7f75a0002840 legacy=0x7f75a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 client.4404.objecter ms_handle_connect 0x7f75a0002840 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 client.4404.objecter resend_mon_ops 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805662211 0 0) 0x7f75a0006590 con 0x7f75a0002840 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 monclient(hunting): my global_id is 4421 2024-09-18T11:44:30.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc122570 con 0x7f75a0002840 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3716462384 0 0) 0x7f75bc122570 con 0x7f75a0002840 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75a0006590 con 0x7f75a0002840 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75ac005030 con 0x7f75a0002840 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 monclient: dump: 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 10 client.4404.objecter ms_handle_connect 0x7f759c0384b0 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3284812840 0 0) 0x7f75a0006590 con 0x7f75a0002840 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.730+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75a000a140 con 0x7f75a0002840 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.734+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f75ac005190 con 0x7f75a0002840 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.734+0000 7f75b2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.734+0000 7f75b2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.734+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6790/0 conn(0x7f75a0002840 legacy=0x7f75a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.734+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6790/0 conn(0x7f75a0002840 legacy=0x7f75a0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:30.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75c1979700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f75a0009670 con 0x7f75a00090e0 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75baffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6790/0 conn(0x7f75a00090e0 legacy=0x7f75a0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75b2ffd700 10 client.4404.objecter ms_handle_connect 0x7f75a00090e0 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75b2ffd700 10 client.4404.objecter resend_mon_ops 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978646552 0 0) 0x7f75a0009670 con 0x7f75a00090e0 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.742+0000 7f75b2ffd700 10 monclient(hunting): my global_id is 4424 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a0006590 con 0x7f75a00090e0 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 547803759 0 0) 0x7f75a0006590 con 0x7f75a00090e0 2024-09-18T11:44:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75a0009670 con 0x7f75a00090e0 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75ac0057b0 con 0x7f75a00090e0 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 10 monclient: dump: 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.759 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3433380283 0 0) 0x7f75a0009670 con 0x7f75a00090e0 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f75a0009930 con 0x7f75a00090e0 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 2775985464) 0x7f75ac005b40 con 0x7f75a00090e0 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6790/0 conn(0x7f75a00090e0 legacy=0x7f75a0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75b2ffd700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6790/0 conn(0x7f75a00090e0 legacy=0x7f75a0008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 8, 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:44:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 21, 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.762 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.763 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:30.764 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.765 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:44:30.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.772 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.773 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:44:30.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.773 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:44:30.773 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:44:30.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 10 librados: watch_flush enter 2024-09-18T11:44:30.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 10 librados: watch_flush exit 2024-09-18T11:44:30.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 20 client.4404.objecter shutdown clearing up homeless session... 2024-09-18T11:44:30.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 10 client.4404.objecter successfully canceled tick 2024-09-18T11:44:30.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.129:6812/20806 conn(0x7f759c0384b0 legacy=0x7f759c03a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 10 monclient: shutdown 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75bb7fe700 1 -- 172.21.15.112:0/1616084626 reap_dead start 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 >> v1:172.21.15.112:6789/0 conn(0x7f75bc0a0570 legacy=0x7f75bc146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 shutdown_connections 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 >> 172.21.15.112:0/1616084626 conn(0x7f75bc067680 msgr2=0x7f75bc0a4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 shutdown_connections 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 1 -- 172.21.15.112:0/1616084626 wait complete. 2024-09-18T11:44:30.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.746+0000 7f75c1979700 1 librados: shutdown 2024-09-18T11:44:30.777 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:44:30.777 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:44:30.777 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:44:30.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient: get_monmap_and_config 2024-09-18T11:44:30.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient: build_initial_monmap 2024-09-18T11:44:30.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient: monmap: 2024-09-18T11:44:30.894 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:30.894 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.894 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:30.894481+0000 2024-09-18T11:44:30.894 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:30.894481+0000 2024-09-18T11:44:30.895 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:30.895 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.895 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:30.895 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:30.895 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:30.895 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 1 Processor -- start 2024-09-18T11:44:30.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 1 -- start start 2024-09-18T11:44:30.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient: init 2024-09-18T11:44:30.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient(hunting): picked mon.noname-c con 0x7fd3f4130420 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient(hunting): picked mon.noname-a con 0x7fd3f406ea60 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient(hunting): picked mon.noname-b con 0x7fd3f4136aa0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f4130f00 con 0x7fd3f406ea60 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f406c990 con 0x7fd3f4130420 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f40a12c0 con 0x7fd3f4136aa0 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3fa385700 10 monclient(hunting): authenticate will time out at 842.735535s 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3f37fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd3f406ea60 0x7fd3f4135380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36742/0 (socket says 172.21.15.112:36742) 2024-09-18T11:44:30.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.890+0000 7fd3f37fe700 1 -- 172.21.15.112:0/1372174446 learned_addr learned my addr 172.21.15.112:0/1372174446 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f37fe700 1 -- 172.21.15.112:0/1372174446 >> v1:172.21.15.112:6789/0 conn(0x7fd3f406ea60 legacy=0x7fd3f4135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f8922700 1 -- 172.21.15.112:0/1372174446 >> v1:172.21.15.112:6790/0 conn(0x7fd3f4136aa0 legacy=0x7fd3f4138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f3fff700 1 -- 172.21.15.112:0/1372174446 >> v1:172.21.15.129:6789/0 conn(0x7fd3f4130420 legacy=0x7fd3f41307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064919094 0 0) 0x7fd3f40a12c0 con 0x7fd3f4136aa0 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): my global_id is 4427 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d4003570 con 0x7fd3f4136aa0 2024-09-18T11:44:30.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570843598 0 0) 0x7fd3f4130f00 con 0x7fd3f406ea60 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): my global_id is 4407 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f40a12c0 con 0x7fd3f406ea60 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369329673 0 0) 0x7fd3f406c990 con 0x7fd3f4130420 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient(hunting): my global_id is 4408 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f4130f00 con 0x7fd3f4130420 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179026139 0 0) 0x7fd3d4003570 con 0x7fd3f4136aa0 2024-09-18T11:44:30.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 >> v1:172.21.15.129:6789/0 conn(0x7fd3f4130420 legacy=0x7fd3f41307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 >> v1:172.21.15.112:6789/0 conn(0x7fd3f406ea60 legacy=0x7fd3f4135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 monclient: found mon.noname-b 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f406d7d0 con 0x7fd3f4136aa0 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3fa385700 5 monclient: authenticate success, global_id 4427 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3fa385700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd3ec001dd0 con 0x7fd3f4136aa0 2024-09-18T11:44:30.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: dump: 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.904 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3fa385700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3ec001f30 con 0x7fd3f4136aa0 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3fa385700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 1 -- 172.21.15.112:0/1372174446 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd3ec002420 con 0x7fd3f4136aa0 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3fa385700 4 set_mon_vals no callback set 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.894+0000 7fd3f2ffd700 10 monclient: dump: 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.905 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient: shutdown 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- 172.21.15.112:0/1372174446 >> v1:172.21.15.112:6790/0 conn(0x7fd3f4136aa0 legacy=0x7fd3f4138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- 172.21.15.112:0/1372174446 shutdown_connections 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- 172.21.15.112:0/1372174446 >> 172.21.15.112:0/1372174446 conn(0x7fd3f40680d0 msgr2=0x7fd3f40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- 172.21.15.112:0/1372174446 shutdown_connections 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- 172.21.15.112:0/1372174446 wait complete. 2024-09-18T11:44:30.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient: build_initial_monmap 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient: monmap: 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:30.907 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 librados: starting msgr at 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 librados: starting objecter 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 Processor -- start 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- start start 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 librados: setting wanted keys 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 librados: calling monclient init 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient: init 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:30.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient(hunting): picked mon.noname-a con 0x7fd3f4130420 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient(hunting): picked mon.noname-c con 0x7fd3f4136aa0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient(hunting): picked mon.noname-b con 0x7fd3f4135970 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f409bce0 con 0x7fd3f4130420 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f406b000 con 0x7fd3f4135970 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f411c880 con 0x7fd3f4136aa0 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3fa385700 10 monclient(hunting): authenticate will time out at 842.742686s 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3f3fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd3f4130420 0x7fd3f4142150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36748/0 (socket says 172.21.15.112:36748) 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.898+0000 7fd3f3fff700 1 -- 172.21.15.112:0/3190572025 learned_addr learned my addr 172.21.15.112:0/3190572025 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3f3fff700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.112:6789/0 conn(0x7fd3f4130420 legacy=0x7fd3f4142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 client.?.objecter ms_handle_connect 0x7fd3f4130420 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 client.?.objecter ms_handle_connect 0x7fd3f4136aa0 2024-09-18T11:44:30.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3f37fe700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.112:6790/0 conn(0x7fd3f4136aa0 legacy=0x7fd3f41471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005632364 0 0) 0x7fd3f409bce0 con 0x7fd3f4130420 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3f8922700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6789/0 conn(0x7fd3f4135970 legacy=0x7fd3f414a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): my global_id is 4410 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d8004f10 con 0x7fd3f4130420 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 client.?.objecter ms_handle_connect 0x7fd3f4135970 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918233038 0 0) 0x7fd3f411c880 con 0x7fd3f4136aa0 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): my global_id is 4430 2024-09-18T11:44:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f409bce0 con 0x7fd3f4136aa0 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579183303 0 0) 0x7fd3f406b000 con 0x7fd3f4135970 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): my global_id is 4411 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f411c880 con 0x7fd3f4135970 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2771279756 0 0) 0x7fd3d8004f10 con 0x7fd3f4130420 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3f406b000 con 0x7fd3f4130420 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd3e4001cd0 con 0x7fd3f4130420 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): dump: 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.917 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1836572357 0 0) 0x7fd3f409bce0 con 0x7fd3f4136aa0 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3d8004f10 con 0x7fd3f4136aa0 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd3dc001bf0 con 0x7fd3f4136aa0 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): dump: 2024-09-18T11:44:30.918 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 369618335 0 0) 0x7fd3f411c880 con 0x7fd3f4135970 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3f409bce0 con 0x7fd3f4135970 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd3ec001af0 con 0x7fd3f4135970 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient(hunting): dump: 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2559911205 0 0) 0x7fd3f406b000 con 0x7fd3f4130420 2024-09-18T11:44:30.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.112:6790/0 conn(0x7fd3f4136aa0 legacy=0x7fd3f41471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6789/0 conn(0x7fd3f4135970 legacy=0x7fd3f414a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 monclient: found mon.a 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f406aaf0 con 0x7fd3f4130420 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient: _finish_auth 0 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 5 monclient: authenticate success, global_id 4410 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 10 monclient: _renew_subs 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f406b410 con 0x7fd3f4130420 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 10 monclient: _renew_subs 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f411d420 con 0x7fd3f4130420 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3fa385700 1 librados: init done 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3e40022f0 con 0x7fd3f4130420 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd3e4002680 con 0x7fd3f4130420 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:30.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3f1ffb700 4 set_mon_vals no callback set 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.902+0000 7fd3ebfff700 10 monclient: dump: 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3fa385700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fd3e40027e0 con 0x7fd3f4130420 2024-09-18T11:44:30.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd3cc006590 con 0x7fd3cc002840 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fd3e40437a0 con 0x7fd3f4130420 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 client.4410.objecter ms_dispatch 0x7fd3f4000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 3 client.4410.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 3 client.4410.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 20 client.4410.objecter dump_active .. 0 homeless 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3f8922700 10 monclient: get_auth_request con 0x7fd3d8038600 auth_method 0 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 client.4410.objecter ms_handle_connect 0x7fd3cc002840 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 client.4410.objecter resend_mon_ops 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3f37fe700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6789/0 conn(0x7fd3cc002840 legacy=0x7fd3cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464583605 0 0) 0x7fd3cc006590 con 0x7fd3cc002840 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 monclient(hunting): my global_id is 4414 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f406b000 con 0x7fd3cc002840 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 client.4410.objecter ms_handle_connect 0x7fd3d8038600 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3094703336 0 0) 0x7fd3f406b000 con 0x7fd3cc002840 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3cc006590 con 0x7fd3cc002840 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd3dc004ef0 con 0x7fd3cc002840 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.906+0000 7fd3ebfff700 10 monclient: dump: 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.925 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.910+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4114569271 0 0) 0x7fd3cc006590 con 0x7fd3cc002840 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.910+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3cc00a140 con 0x7fd3cc002840 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.910+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fd3dc005920 con 0x7fd3cc002840 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.910+0000 7fd3ebfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.910+0000 7fd3ebfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.910+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6789/0 conn(0x7fd3cc002840 legacy=0x7fd3cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.910+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6789/0 conn(0x7fd3cc002840 legacy=0x7fd3cc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:30.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.922+0000 7fd3fa385700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:44:30.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.922+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd3cc009650 con 0x7fd3cc0090c0 2024-09-18T11:44:30.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3f37fe700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6789/0 conn(0x7fd3cc0090c0 legacy=0x7fd3cc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:30.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 client.4410.objecter ms_handle_connect 0x7fd3cc0090c0 2024-09-18T11:44:30.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 client.4410.objecter resend_mon_ops 2024-09-18T11:44:30.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963363903 0 0) 0x7fd3cc009650 con 0x7fd3cc0090c0 2024-09-18T11:44:30.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:30.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:30.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 monclient(hunting): my global_id is 4417 2024-09-18T11:44:30.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3cc006590 con 0x7fd3cc0090c0 2024-09-18T11:44:30.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2669444827 0 0) 0x7fd3cc006590 con 0x7fd3cc0090c0 2024-09-18T11:44:30.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3cc009650 con 0x7fd3cc0090c0 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd3dc004d90 con 0x7fd3cc0090c0 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 monclient: dump: 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:30.930 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:30.931 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:30.931 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:30.931 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:30.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4284281638 0 0) 0x7fd3cc009650 con 0x7fd3cc0090c0 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd3cc009910 con 0x7fd3cc0090c0 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 1854867318) 0x7fd3dc0055d0 con 0x7fd3cc0090c0 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6789/0 conn(0x7fd3cc0090c0 legacy=0x7fd3cc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3ebfff700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6789/0 conn(0x7fd3cc0090c0 legacy=0x7fd3cc008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:44:30.932 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 8, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 21, 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:30.933 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.934 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.935 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:30.936 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:30.937 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.938 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.940 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:44:30.941 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.943 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:44:30.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:44:30.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:44:30.946 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 10 librados: watch_flush enter 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 10 librados: watch_flush exit 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 20 client.4410.objecter shutdown clearing up homeless session... 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 10 client.4410.objecter successfully canceled tick 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.129:6812/20806 conn(0x7fd3d8038600 legacy=0x7fd3d803aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 10 monclient: shutdown 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 >> v1:172.21.15.112:6789/0 conn(0x7fd3f4130420 legacy=0x7fd3f4142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3f8922700 1 -- 172.21.15.112:0/3190572025 reap_dead start 2024-09-18T11:44:30.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3f8922700 1 -- 172.21.15.112:0/3190572025 reap_dead start 2024-09-18T11:44:30.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 shutdown_connections 2024-09-18T11:44:30.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 >> 172.21.15.112:0/3190572025 conn(0x7fd3f40680d0 msgr2=0x7fd3f409aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:30.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 shutdown_connections 2024-09-18T11:44:30.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 1 -- 172.21.15.112:0/3190572025 wait complete. 2024-09-18T11:44:30.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:30.926+0000 7fd3fa385700 1 librados: shutdown 2024-09-18T11:44:30.951 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:44:30.952 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T11:44:30.952 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T11:44:30.952 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T11:44:30.952 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-18T11:44:30.952 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-18T11:44:31.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient: get_monmap_and_config 2024-09-18T11:44:31.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient: build_initial_monmap 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient: monmap: 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:31.066862+0000 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:31.066862+0000 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:31.067 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 1 Processor -- start 2024-09-18T11:44:31.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 1 -- start start 2024-09-18T11:44:31.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient: init 2024-09-18T11:44:31.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:31.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:31.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient(hunting): picked mon.noname-a con 0x7fdd3406f710 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:31.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient(hunting): picked mon.noname-b con 0x7fdd3409b7c0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:31.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient(hunting): picked mon.noname-c con 0x7fdd3409d960 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:31.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd340955a0 con 0x7fdd3406f710 2024-09-18T11:44:31.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3409b0c0 con 0x7fdd3409d960 2024-09-18T11:44:31.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3409ade0 con 0x7fdd3409b7c0 2024-09-18T11:44:31.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:31.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.062+0000 7fdd3a8a9700 10 monclient(hunting): authenticate will time out at 842.907833s 2024-09-18T11:44:31.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd390a6700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fdd3409b7c0 0x7fdd340991c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36554/0 (socket says 172.21.15.112:36554) 2024-09-18T11:44:31.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd390a6700 1 -- 172.21.15.112:0/2444062440 learned_addr learned my addr 172.21.15.112:0/2444062440 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:31.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd398a7700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdd3406f710 0x7fdd34098ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36754/0 (socket says 172.21.15.112:36754) 2024-09-18T11:44:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd390a6700 1 -- 172.21.15.112:0/2444062440 >> v1:172.21.15.112:6790/0 conn(0x7fdd3409b7c0 legacy=0x7fdd340991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd3a0a8700 1 -- 172.21.15.112:0/2444062440 >> v1:172.21.15.129:6789/0 conn(0x7fdd3409d960 legacy=0x7fdd34137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd390a6700 1 -- 172.21.15.112:0/2444062440 >> v1:172.21.15.112:6790/0 conn(0x7fdd3409b7c0 legacy=0x7fdd340991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-18T11:44:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd390a6700 1 -- 172.21.15.112:0/2444062440 >> v1:172.21.15.112:6790/0 conn(0x7fdd3409b7c0 legacy=0x7fdd340991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:44:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd390a6700 1 --1- 172.21.15.112:0/2444062440 >> v1:172.21.15.112:6790/0 conn(0x7fdd3409b7c0 0x7fdd340991c0 :-1 s=OPENED pgs=134 cs=1 l=1).handle_message read tag failed 2024-09-18T11:44:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd390a6700 1 --1- 172.21.15.112:0/2444062440 >> v1:172.21.15.112:6790/0 conn(0x7fdd3409b7c0 0x7fdd340991c0 :-1 s=OPENED pgs=134 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:44:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd398a7700 1 -- 172.21.15.112:0/2444062440 >> v1:172.21.15.112:6789/0 conn(0x7fdd3406f710 legacy=0x7fdd34098ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6790/0 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628151883 0 0) 0x7fdd3409b0c0 con 0x7fdd3409d960 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient(hunting): my global_id is 4420 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd1c003570 con 0x7fdd3409d960 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632263232 0 0) 0x7fdd340955a0 con 0x7fdd3406f710 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient(hunting): my global_id is 4413 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd3409b0c0 con 0x7fdd3406f710 2024-09-18T11:44:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2981568375 0 0) 0x7fdd3409b0c0 con 0x7fdd3406f710 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 >> v1:172.21.15.112:6790/0 conn(0x7fdd3409b7c0 legacy=0x7fdd340991c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 >> v1:172.21.15.129:6789/0 conn(0x7fdd3409d960 legacy=0x7fdd34137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: _finish_hunting 0 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 monclient: found mon.noname-a 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd340998d0 con 0x7fdd3406f710 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: _finish_auth 0 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdd280018e0 con 0x7fdd3406f710 2024-09-18T11:44:31.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:44:31.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: dump: 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.079 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd3a8a9700 5 monclient: authenticate success, global_id 4413 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd3a8a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd28001b70 con 0x7fdd3406f710 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 1 -- 172.21.15.112:0/2444062440 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdd28004830 con 0x7fdd3406f710 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd388a5700 10 monclient: dump: 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.080 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd3a8a9700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.066+0000 7fdd3a8a9700 4 set_mon_vals no callback set 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient: shutdown 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2444062440 >> v1:172.21.15.112:6789/0 conn(0x7fdd3406f710 legacy=0x7fdd34098ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2444062440 shutdown_connections 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2444062440 >> 172.21.15.112:0/2444062440 conn(0x7fdd34068210 msgr2=0x7fdd340a15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:31.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2444062440 shutdown_connections 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2444062440 wait complete. 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient: build_initial_monmap 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient: monmap: 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 librados: starting msgr at 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 librados: starting objecter 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 Processor -- start 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- start start 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 librados: setting wanted keys 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 librados: calling monclient init 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient: init 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient(hunting): picked mon.noname-a con 0x7fdd3409b7c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:31.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient(hunting): picked mon.noname-c con 0x7fdd3409d960 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:31.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient(hunting): picked mon.noname-b con 0x7fdd3408eb30 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:31.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3406fbe0 con 0x7fdd3409b7c0 2024-09-18T11:44:31.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd34142670 con 0x7fdd3408eb30 2024-09-18T11:44:31.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3408fa80 con 0x7fdd3409d960 2024-09-18T11:44:31.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:31.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd3a8a9700 10 monclient(hunting): authenticate will time out at 842.914800s 2024-09-18T11:44:31.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd390a6700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fdd3409d960 0x7fdd34141f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36558/0 (socket says 172.21.15.112:36558) 2024-09-18T11:44:31.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.070+0000 7fdd390a6700 1 -- 172.21.15.112:0/2473346712 learned_addr learned my addr 172.21.15.112:0/2473346712 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:31.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd390a6700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6790/0 conn(0x7fdd3409d960 legacy=0x7fdd34141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 client.?.objecter ms_handle_connect 0x7fdd3409d960 2024-09-18T11:44:31.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:31.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16745679 0 0) 0x7fdd3408fa80 con 0x7fdd3409d960 2024-09-18T11:44:31.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd398a7700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6789/0 conn(0x7fdd3409b7c0 legacy=0x7fdd34141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): my global_id is 4433 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd20004f10 con 0x7fdd3409d960 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 client.?.objecter ms_handle_connect 0x7fdd3409b7c0 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 client.?.objecter ms_handle_connect 0x7fdd3408eb30 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a0a8700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.129:6789/0 conn(0x7fdd3408eb30 legacy=0x7fdd34149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216493958 0 0) 0x7fdd3406fbe0 con 0x7fdd3409b7c0 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): my global_id is 4416 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd3408fa80 con 0x7fdd3409b7c0 2024-09-18T11:44:31.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3526408225 0 0) 0x7fdd20004f10 con 0x7fdd3409d960 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd3406fbe0 con 0x7fdd3409d960 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdd2c002080 con 0x7fdd3409d960 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): dump: 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.091 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266530236 0 0) 0x7fdd34142670 con 0x7fdd3408eb30 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient(hunting): my global_id is 4423 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd20004f10 con 0x7fdd3408eb30 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1232182948 0 0) 0x7fdd3406fbe0 con 0x7fdd3409d960 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.129:6789/0 conn(0x7fdd3408eb30 legacy=0x7fdd34149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6789/0 conn(0x7fdd3409b7c0 legacy=0x7fdd34141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 monclient: found mon.c 2024-09-18T11:44:31.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd340a14f0 con 0x7fdd3409d960 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 5 monclient: authenticate success, global_id 4433 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 10 monclient: _renew_subs 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd340923c0 con 0x7fdd3409d960 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 10 monclient: _renew_subs 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd34090140 con 0x7fdd3409d960 2024-09-18T11:44:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd3a8a9700 1 librados: init done 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd2c004f60 con 0x7fdd3409d960 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdd2c0052f0 con 0x7fdd3409d960 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd33fff700 4 set_mon_vals no callback set 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.074+0000 7fdd31ffb700 10 monclient: dump: 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.094 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd3a8a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd14006590 con 0x7fdd14002840 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fdd2c005550 con 0x7fdd3409d960 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fdd2c043c60 con 0x7fdd3409d960 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 client.4433.objecter ms_dispatch 0x7fdd34000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 3 client.4433.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 3 client.4433.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 20 client.4433.objecter dump_active .. 0 homeless 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd3a0a8700 10 monclient: get_auth_request con 0x7fdd200381e0 auth_method 0 2024-09-18T11:44:31.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 client.4433.objecter ms_handle_connect 0x7fdd14002840 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd398a7700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6789/0 conn(0x7fdd14002840 legacy=0x7fdd14004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 client.4433.objecter resend_mon_ops 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132312704 0 0) 0x7fdd14006590 con 0x7fdd14002840 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 monclient(hunting): my global_id is 4419 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd3406fbe0 con 0x7fdd14002840 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3611713775 0 0) 0x7fdd3406fbe0 con 0x7fdd14002840 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd14006590 con 0x7fdd14002840 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdd28004ce0 con 0x7fdd14002840 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:31.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 monclient: dump: 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 10 client.4433.objecter ms_handle_connect 0x7fdd200381e0 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1890281787 0 0) 0x7fdd14006590 con 0x7fdd14002840 2024-09-18T11:44:31.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.078+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd1400a140 con 0x7fdd14002840 2024-09-18T11:44:31.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.082+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fdd28006fc0 con 0x7fdd14002840 2024-09-18T11:44:31.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.082+0000 7fdd31ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:31.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.082+0000 7fdd31ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:31.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.082+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6789/0 conn(0x7fdd14002840 legacy=0x7fdd14004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.082+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6789/0 conn(0x7fdd14002840 legacy=0x7fdd14004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:31.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.090+0000 7fdd3a8a9700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:44:31.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.090+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd14009690 con 0x7fdd14009100 2024-09-18T11:44:31.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd398a7700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6789/0 conn(0x7fdd14009100 legacy=0x7fdd14008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 10 client.4433.objecter ms_handle_connect 0x7fdd14009100 2024-09-18T11:44:31.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 10 client.4433.objecter resend_mon_ops 2024-09-18T11:44:31.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915370160 0 0) 0x7fdd14009690 con 0x7fdd14009100 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 10 monclient(hunting): my global_id is 4422 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd14006590 con 0x7fdd14009100 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3986560796 0 0) 0x7fdd14006590 con 0x7fdd14009100 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd14009690 con 0x7fdd14009100 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdd28005410 con 0x7fdd14009100 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 10 monclient: dump: 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.100 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.101 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2551502470 0 0) 0x7fdd14009690 con 0x7fdd14009100 2024-09-18T11:44:31.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.094+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7fdd14009950 con 0x7fdd14009100 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7fdd28005910 con 0x7fdd14009100 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd31ffb700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd31ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6789/0 conn(0x7fdd14009100 legacy=0x7fdd14008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd31ffb700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6789/0 conn(0x7fdd14009100 legacy=0x7fdd14008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: "ret": 0, 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-18T11:44:31.102 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 10 librados: watch_flush enter 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 10 librados: watch_flush exit 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 20 client.4433.objecter shutdown clearing up homeless session... 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 10 client.4433.objecter successfully canceled tick 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.129:6812/20806 conn(0x7fdd200381e0 legacy=0x7fdd2003a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 10 monclient: shutdown 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 >> v1:172.21.15.112:6790/0 conn(0x7fdd3409d960 legacy=0x7fdd34141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a0a8700 1 -- 172.21.15.112:0/2473346712 reap_dead start 2024-09-18T11:44:31.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a0a8700 1 -- 172.21.15.112:0/2473346712 reap_dead start 2024-09-18T11:44:31.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 shutdown_connections 2024-09-18T11:44:31.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 >> 172.21.15.112:0/2473346712 conn(0x7fdd34068210 msgr2=0x7fdd3409c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:31.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 shutdown_connections 2024-09-18T11:44:31.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 1 -- 172.21.15.112:0/2473346712 wait complete. 2024-09-18T11:44:31.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.098+0000 7fdd3a8a9700 1 librados: shutdown 2024-09-18T11:44:31.119 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T11:44:31.120 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T11:44:31.120 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:44:31.132 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T11:44:31.132 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T11:44:31.132 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:44:31.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient: get_monmap_and_config 2024-09-18T11:44:31.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient: build_initial_monmap 2024-09-18T11:44:31.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient: monmap: 2024-09-18T11:44:31.237 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:31.237 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.237 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:31.237439+0000 2024-09-18T11:44:31.238 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:31.237439+0000 2024-09-18T11:44:31.238 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:31.238 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.238 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:31.238 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:31.238 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:31.238 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 1 Processor -- start 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 1 -- start start 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient: init 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f9c0c09aaf0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f9c0c068ee0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f9c0c096350 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0983b0 con 0x7f9c0c068ee0 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0a01d0 con 0x7f9c0c09aaf0 2024-09-18T11:44:31.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0953d0 con 0x7f9c0c096350 2024-09-18T11:44:31.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:31.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0bfff700 10 monclient(hunting): authenticate will time out at 843.078369s 2024-09-18T11:44:31.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0a7fc700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f9c0c068ee0 legacy=0x7f9c0c067d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:31.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0b7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f9c0c096350 0x7f9c0c1385c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36586/0 (socket says 172.21.15.112:36586) 2024-09-18T11:44:31.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.234+0000 7f9c0b7fe700 1 -- 172.21.15.112:0/656838238 learned_addr learned my addr 172.21.15.112:0/656838238 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:31.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0b7fe700 1 -- 172.21.15.112:0/656838238 >> v1:172.21.15.112:6790/0 conn(0x7f9c0c096350 legacy=0x7f9c0c1385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664800790 0 0) 0x7f9c0c0953d0 con 0x7f9c0c096350 2024-09-18T11:44:31.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient(hunting): my global_id is 4436 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4003570 con 0x7f9c0c096350 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0affd700 1 -- 172.21.15.112:0/656838238 >> v1:172.21.15.129:6789/0 conn(0x7f9c0c09aaf0 legacy=0x7f9c0c067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2231432598 0 0) 0x7f9bf4003570 con 0x7f9c0c096350 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 >> v1:172.21.15.129:6789/0 conn(0x7f9c0c09aaf0 legacy=0x7f9c0c067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 >> v1:172.21.15.112:6789/0 conn(0x7f9c0c068ee0 legacy=0x7f9c0c067d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 monclient: found mon.noname-b 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c09aec0 con 0x7f9c0c096350 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: _finish_auth 0 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:31.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9c04001940 con 0x7f9c0c096350 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 5 monclient: authenticate success, global_id 4436 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: dump: 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.246 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c04001c10 con 0x7f9c0c096350 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 1 -- 172.21.15.112:0/656838238 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9c04004a90 con 0x7f9c0c096350 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c09ffb700 10 monclient: dump: 2024-09-18T11:44:31.247 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 4 set_mon_vals no callback set 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 10 monclient: shutdown 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 1 -- 172.21.15.112:0/656838238 >> v1:172.21.15.112:6790/0 conn(0x7f9c0c096350 legacy=0x7f9c0c1385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 1 -- 172.21.15.112:0/656838238 shutdown_connections 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 1 -- 172.21.15.112:0/656838238 >> 172.21.15.112:0/656838238 conn(0x7f9c0c0a1f90 msgr2=0x7f9c0c09f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 1 -- 172.21.15.112:0/656838238 shutdown_connections 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 1 -- 172.21.15.112:0/656838238 wait complete. 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 10 monclient: build_initial_monmap 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 10 monclient: monmap: 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:31.249 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 1 librados: starting msgr at 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 1 librados: starting objecter 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.238+0000 7f9c0bfff700 1 Processor -- start 2024-09-18T11:44:31.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 1 -- start start 2024-09-18T11:44:31.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 1 librados: setting wanted keys 2024-09-18T11:44:31.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 1 librados: calling monclient init 2024-09-18T11:44:31.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 10 monclient: init 2024-09-18T11:44:31.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f9c0c068ee0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f9c0c096350 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f9c0c09aaf0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c091680 con 0x7f9c0c09aaf0 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c090540 con 0x7f9c0c096350 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0913a0 con 0x7f9c0c068ee0 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0bfff700 10 monclient(hunting): authenticate will time out at 843.084960s 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0b7fe700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f9c0c09aaf0 legacy=0x7f9c0c099700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0affd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f9c0c068ee0 0x7f9c0c0988e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36588/0 (socket says 172.21.15.112:36588) 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0affd700 1 -- 172.21.15.112:0/3502843754 learned_addr learned my addr 172.21.15.112:0/3502843754 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0affd700 1 -- 172.21.15.112:0/3502843754 >> v1:172.21.15.112:6790/0 conn(0x7f9c0c068ee0 legacy=0x7f9c0c0988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 10 client.?.objecter ms_handle_connect 0x7f9c0c068ee0 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276607553 0 0) 0x7f9c0c0913a0 con 0x7f9c0c068ee0 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 10 monclient(hunting): my global_id is 4439 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf8004f10 con 0x7f9c0c068ee0 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 10 client.?.objecter ms_handle_connect 0x7f9c0c096350 2024-09-18T11:44:31.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9bf2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.242+0000 7f9c0a7fc700 1 -- 172.21.15.112:0/3502843754 >> v1:172.21.15.129:6789/0 conn(0x7f9c0c096350 legacy=0x7f9c0c098ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226266117 0 0) 0x7f9c0c090540 con 0x7f9c0c096350 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient(hunting): my global_id is 4429 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c0c0913a0 con 0x7f9c0c096350 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2964573805 0 0) 0x7f9bf8004f10 con 0x7f9c0c068ee0 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c0c090540 con 0x7f9c0c068ee0 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9c00001c90 con 0x7f9c0c068ee0 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient(hunting): dump: 2024-09-18T11:44:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4246433065 0 0) 0x7f9c0c090540 con 0x7f9c0c068ee0 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 >> v1:172.21.15.129:6789/0 conn(0x7f9c0c096350 legacy=0x7f9c0c098ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 >> v1:172.21.15.112:6789/0 conn(0x7f9c0c09aaf0 legacy=0x7f9c0c099700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 monclient: found mon.c 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c091960 con 0x7f9c0c068ee0 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient: _finish_auth 0 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 5 monclient: authenticate success, global_id 4439 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 10 monclient: _renew_subs 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c090e40 con 0x7f9c0c068ee0 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 10 monclient: _renew_subs 2024-09-18T11:44:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c095070 con 0x7f9c0c068ee0 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 1 librados: init done 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c000022b0 con 0x7f9c0c068ee0 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9c000025c0 con 0x7f9c0c068ee0 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 monclient: dump: 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c097fa700 4 set_mon_vals no callback set 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f9c00002720 con 0x7f9c0c068ee0 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec0050b0 con 0x7f9c0c068ee0 2024-09-18T11:44:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7f9c00043790 con 0x7f9c0c068ee0 2024-09-18T11:44:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 10 client.4439.objecter ms_dispatch 0x7f9c0c000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 3 client.4439.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 3 client.4439.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9bf2ffd700 20 client.4439.objecter dump_active .. 0 homeless 2024-09-18T11:44:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.246+0000 7f9c0a7fc700 10 monclient: get_auth_request con 0x7f9bf8037ea0 auth_method 0 2024-09-18T11:44:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.250+0000 7f9bf2ffd700 10 client.4439.objecter ms_handle_connect 0x7f9bf8037ea0 2024-09-18T11:44:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.258+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f9c00002c10 con 0x7f9c0c068ee0 2024-09-18T11:44:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.258+0000 7f9bf2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:31.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.258+0000 7f9bf2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:31.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.482+0000 7f9c0bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:44:31.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.482+0000 7f9c0bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:44:31.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.482+0000 7f9c0bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:31.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.482+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9bec0058e0 con 0x7f9c0c068ee0 2024-09-18T11:44:31.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.482+0000 7f9bf2ffd700 1 -- 172.21.15.112:0/3502843754 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1392720101) 0x7f9c000144a0 con 0x7f9c0c068ee0 2024-09-18T11:44:31.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.482+0000 7f9bf2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:44:31.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.482+0000 7f9bf2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:31.486 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:44:31.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 10 librados: watch_flush enter 2024-09-18T11:44:31.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 10 librados: watch_flush exit 2024-09-18T11:44:31.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 20 client.4439.objecter shutdown clearing up homeless session... 2024-09-18T11:44:31.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 10 client.4439.objecter successfully canceled tick 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 >> v1:172.21.15.129:6812/20806 conn(0x7f9bf8037ea0 legacy=0x7f9bf803a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 10 monclient: shutdown 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 >> v1:172.21.15.112:6790/0 conn(0x7f9c0c068ee0 legacy=0x7f9c0c0988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 shutdown_connections 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 >> 172.21.15.112:0/3502843754 conn(0x7f9c0c0a1f90 msgr2=0x7f9c0c09ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 shutdown_connections 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 1 -- 172.21.15.112:0/3502843754 wait complete. 2024-09-18T11:44:31.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:31.486+0000 7f9c0bfff700 1 librados: shutdown 2024-09-18T11:44:34.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:44:34.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.610+0000 7fe6c1b25700 10 monclient: get_monmap_and_config 2024-09-18T11:44:34.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.610+0000 7fe6c1b25700 10 monclient: build_initial_monmap 2024-09-18T11:44:34.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient: monmap: 2024-09-18T11:44:34.616 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:34.616 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:34.616 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:34.616414+0000 2024-09-18T11:44:34.616 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:34.616414+0000 2024-09-18T11:44:34.616 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:34.617 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:34.617 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:34.617 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:34.617 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:34.617 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 1 Processor -- start 2024-09-18T11:44:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 1 -- start start 2024-09-18T11:44:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient: init 2024-09-18T11:44:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient(hunting): picked mon.noname-a con 0x7fe6bc06e0e0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient(hunting): picked mon.noname-c con 0x7fe6bc06e4b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:34.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient(hunting): picked mon.noname-b con 0x7fe6bc138520 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:34.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc09fee0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc09fc00 con 0x7fe6bc06e4b0 2024-09-18T11:44:34.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc0a00c0 con 0x7fe6bc138520 2024-09-18T11:44:34.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:34.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6bb7fe700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe6bc06e0e0 legacy=0x7fe6bc06c990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:34.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6c1b25700 10 monclient(hunting): authenticate will time out at 846.457715s 2024-09-18T11:44:34.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6bbfff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe6bc138520 0x7fe6bc13a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36590/0 (socket says 172.21.15.112:36590) 2024-09-18T11:44:34.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.614+0000 7fe6bbfff700 1 -- 172.21.15.112:0/3716047508 learned_addr learned my addr 172.21.15.112:0/3716047508 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:34.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6bbfff700 1 -- 172.21.15.112:0/3716047508 >> v1:172.21.15.112:6790/0 conn(0x7fe6bc138520 legacy=0x7fe6bc13a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:34.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6baffd700 1 -- 172.21.15.112:0/3716047508 >> v1:172.21.15.129:6789/0 conn(0x7fe6bc06e4b0 legacy=0x7fe6bc06e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:34.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314999180 0 0) 0x7fe6bc0a00c0 con 0x7fe6bc138520 2024-09-18T11:44:34.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:34.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient(hunting): my global_id is 4442 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a8003570 con 0x7fe6bc138520 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296247833 0 0) 0x7fe6bc09fc00 con 0x7fe6bc06e4b0 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient(hunting): my global_id is 4432 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6bc0a00c0 con 0x7fe6bc06e4b0 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787734809 0 0) 0x7fe6a8003570 con 0x7fe6bc138520 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 >> v1:172.21.15.129:6789/0 conn(0x7fe6bc06e4b0 legacy=0x7fe6bc06e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 >> v1:172.21.15.112:6789/0 conn(0x7fe6bc06e0e0 legacy=0x7fe6bc06c990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 monclient: found mon.noname-b 2024-09-18T11:44:34.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc06b5c0 con 0x7fe6bc138520 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6b4001a00 con 0x7fe6bc138520 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 5 monclient: authenticate success, global_id 4442 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: dump: 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:34.623 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6b4001be0 con 0x7fe6bc138520 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 1 -- 172.21.15.112:0/3716047508 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6b4004a90 con 0x7fe6bc138520 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:34.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 4 set_mon_vals no callback set 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6ba7fc700 10 monclient: dump: 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:34.625 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:34.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 10 monclient: shutdown 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3716047508 >> v1:172.21.15.112:6790/0 conn(0x7fe6bc138520 legacy=0x7fe6bc13a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3716047508 shutdown_connections 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3716047508 >> 172.21.15.112:0/3716047508 conn(0x7fe6bc068100 msgr2=0x7fe6bc0a3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3716047508 shutdown_connections 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3716047508 wait complete. 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 10 monclient: build_initial_monmap 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 10 monclient: monmap: 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:34.634 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 1 librados: starting msgr at 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 1 librados: starting objecter 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.618+0000 7fe6c1b25700 1 Processor -- start 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 1 -- start start 2024-09-18T11:44:34.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 1 librados: setting wanted keys 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 1 librados: calling monclient init 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 10 monclient: init 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 10 monclient(hunting): picked mon.noname-c con 0x7fe6bc06e0e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 10 monclient(hunting): picked mon.noname-a con 0x7fe6bc06e4b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 10 monclient(hunting): picked mon.noname-b con 0x7fe6bc138520 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc1369b0 con 0x7fe6bc06e4b0 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc136c90 con 0x7fe6bc138520 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc136f70 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:34.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6baffd700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe6bc06e4b0 legacy=0x7fe6bc1312c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6c1b25700 10 monclient(hunting): authenticate will time out at 846.464759s 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6bb7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe6bc06e0e0 0x7fe6bc1379a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36596/0 (socket says 172.21.15.112:36596) 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6bb7fe700 1 -- 172.21.15.112:0/3188380051 learned_addr learned my addr 172.21.15.112:0/3188380051 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6bb7fe700 1 -- 172.21.15.112:0/3188380051 >> v1:172.21.15.112:6790/0 conn(0x7fe6bc06e0e0 legacy=0x7fe6bc1379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 client.?.objecter ms_handle_connect 0x7fe6bc06e0e0 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373020882 0 0) 0x7fe6bc136f70 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6bbfff700 1 -- 172.21.15.112:0/3188380051 >> v1:172.21.15.129:6789/0 conn(0x7fe6bc138520 legacy=0x7fe6bc1319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 monclient(hunting): my global_id is 4445 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe69c004f10 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 client.?.objecter ms_handle_connect 0x7fe6bc138520 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820972520 0 0) 0x7fe6bc136c90 con 0x7fe6bc138520 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 10 monclient(hunting): my global_id is 4435 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6bc136f70 con 0x7fe6bc138520 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1506792760 0 0) 0x7fe69c004f10 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.622+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6bc136c90 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6a40045a0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:34.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient(hunting): dump: 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3796217078 0 0) 0x7fe6bc136c90 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 >> v1:172.21.15.129:6789/0 conn(0x7fe6bc138520 legacy=0x7fe6bc1319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 >> v1:172.21.15.112:6789/0 conn(0x7fe6bc06e4b0 legacy=0x7fe6bc1312c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 monclient: found mon.c 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc135ac0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 5 monclient: authenticate success, global_id 4445 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 10 monclient: _renew_subs 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6bc135d10 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 10 monclient: _renew_subs 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6bc14ab80 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 1 librados: init done 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6a4004780 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6a4004ad0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b9ffb700 4 set_mon_vals no callback set 2024-09-18T11:44:34.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 monclient: dump: 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:34.642 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a00050b0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe6a4004ce0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fe6a4043740 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 10 client.4445.objecter ms_dispatch 0x7fe6bc000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 3 client.4445.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 3 client.4445.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6b37fe700 20 client.4445.objecter dump_active .. 0 homeless 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.626+0000 7fe6baffd700 10 monclient: get_auth_request con 0x7fe69c038040 auth_method 0 2024-09-18T11:44:34.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.630+0000 7fe6b37fe700 10 client.4445.objecter ms_handle_connect 0x7fe69c038040 2024-09-18T11:44:34.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.634+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fe6a4014ab0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.634+0000 7fe6b37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:34.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.634+0000 7fe6b37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:34.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:44:34.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:44:34.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:34.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe6a00058e0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6b37fe700 1 -- 172.21.15.112:0/3188380051 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3438780124) 0x7fe6a40144b0 con 0x7fe6bc06e0e0 2024-09-18T11:44:34.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6b37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:44:34.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6b37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:34.887 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:44:34.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 10 librados: watch_flush enter 2024-09-18T11:44:34.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 10 librados: watch_flush exit 2024-09-18T11:44:34.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 20 client.4445.objecter shutdown clearing up homeless session... 2024-09-18T11:44:34.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 10 client.4445.objecter successfully canceled tick 2024-09-18T11:44:34.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 >> v1:172.21.15.129:6812/20806 conn(0x7fe69c038040 legacy=0x7fe69c03a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:34.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 10 monclient: shutdown 2024-09-18T11:44:34.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:34.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 >> v1:172.21.15.112:6790/0 conn(0x7fe6bc06e0e0 legacy=0x7fe6bc1379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:34.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 shutdown_connections 2024-09-18T11:44:34.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 >> 172.21.15.112:0/3188380051 conn(0x7fe6bc068100 msgr2=0x7fe6bc0a2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:34.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 shutdown_connections 2024-09-18T11:44:34.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 1 -- 172.21.15.112:0/3188380051 wait complete. 2024-09-18T11:44:34.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:34.862+0000 7fe6c1b25700 1 librados: shutdown 2024-09-18T11:44:37.892 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient: get_monmap_and_config 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient: build_initial_monmap 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient: monmap: 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:38.008614+0000 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:38.008614+0000 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:38.009 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:38.010 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:38.010 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:38.010 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:38.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 1 Processor -- start 2024-09-18T11:44:38.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 1 -- start start 2024-09-18T11:44:38.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient: init 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f04700a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f04700a0750 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f04701383a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0470096f90 con 0x7f04701383a0 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04700a1d90 con 0x7f04700a0380 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0470096cb0 con 0x7f04700a0750 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:38.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046f59e700 10 monclient(hunting): authenticate will time out at 849.849489s 2024-09-18T11:44:38.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046ed9d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f04701383a0 legacy=0x7f047013a780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:38.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046dd9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f04700a0750 0x7f04700a0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:36600/0 (socket says 172.21.15.112:36600) 2024-09-18T11:44:38.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046dd9b700 1 -- 172.21.15.112:0/2112534021 learned_addr learned my addr 172.21.15.112:0/2112534021 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:38.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.006+0000 7f046dd9b700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6790/0 conn(0x7f04700a0750 legacy=0x7f04700a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:38.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.010+0000 7f046e59c700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.129:6789/0 conn(0x7f04700a0380 legacy=0x7f0470097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:38.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.210+0000 7f046ed9d700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f04701383a0 legacy=0x7f047013a780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:38.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:38.614+0000 7f046ed9d700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f04701383a0 legacy=0x7f047013a780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:39.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:39.418+0000 7f046ed9d700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f04701383a0 legacy=0x7f047013a780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:41.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 10 monclient(hunting): tick 2024-09-18T11:44:41.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 1 monclient(hunting): continuing hunt 2024-09-18T11:44:41.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:44:41.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6790/0 conn(0x7f04700a0750 legacy=0x7f04700a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:41.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.129:6789/0 conn(0x7f04700a0380 legacy=0x7f0470097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f04701383a0 legacy=0x7f047013a780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f0450003810 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f04700a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f04700a0750 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0470097270 con 0x7f0450003810 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f046e59c700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450005c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0470096f90 con 0x7f04700a0380 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0450010500 con 0x7f04700a0750 2024-09-18T11:44:41.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.006+0000 7f045ffff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:41.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.010+0000 7f046dd9b700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.129:6789/0 conn(0x7f04700a0380 legacy=0x7f04500094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:41.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.010+0000 7f046ed9d700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6790/0 conn(0x7f04700a0750 legacy=0x7f045000cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.210+0000 7f046e59c700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450005c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:41.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:41.614+0000 7f046e59c700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450005c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:42.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:42.418+0000 7f046e59c700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450005c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 10 monclient(hunting): tick 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 1 monclient(hunting): continuing hunt 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6790/0 conn(0x7f04700a0750 legacy=0x7f045000cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.129:6789/0 conn(0x7f04700a0380 legacy=0x7f04500094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450005c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f0450010840 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f0450003810 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f04700a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0450011760 con 0x7f0450003810 2024-09-18T11:44:44.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0470097270 con 0x7f04700a0380 2024-09-18T11:44:44.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 1 -- 172.21.15.112:0/2112534021 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045000cce0 con 0x7f0450010840 2024-09-18T11:44:44.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f046dd9b700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450006420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:44.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.006+0000 7f045ffff700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:44.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.010+0000 7f046e59c700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6790/0 conn(0x7f0450010840 legacy=0x7f0450010c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:44.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.010+0000 7f046ed9d700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.129:6789/0 conn(0x7f04700a0380 legacy=0x7f0450005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:44.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.210+0000 7f046dd9b700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450006420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:44.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:44.614+0000 7f046dd9b700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450006420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:45.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823834437 0 0) 0x7f045000cce0 con 0x7f0450010840 2024-09-18T11:44:45.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient(hunting): my global_id is 4448 2024-09-18T11:44:45.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0458003570 con 0x7f0450010840 2024-09-18T11:44:45.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2262708206 0 0) 0x7f0458003570 con 0x7f0450010840 2024-09-18T11:44:45.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.129:6789/0 conn(0x7f04700a0380 legacy=0x7f0450005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6789/0 conn(0x7f0450003810 legacy=0x7f0450006420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:44:45.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: _finish_hunting 0 2024-09-18T11:44:45.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 monclient: found mon.noname-b 2024-09-18T11:44:45.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:45.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:45.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045000ce70 con 0x7f0450010840 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: _finish_auth 0 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f04640019b0 con 0x7f0450010840 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046f59e700 5 monclient: authenticate success, global_id 4448 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: dump: 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.272 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0464001c40 con 0x7f0450010840 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 1 -- 172.21.15.112:0/2112534021 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f04640020a0 con 0x7f0450010840 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046f59e700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046f59e700 4 set_mon_vals no callback set 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.266+0000 7f046d59a700 10 monclient: dump: 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.274 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 10 monclient: shutdown 2024-09-18T11:44:45.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 -- 172.21.15.112:0/2112534021 >> v1:172.21.15.112:6790/0 conn(0x7f0450010840 legacy=0x7f0450010c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 -- 172.21.15.112:0/2112534021 shutdown_connections 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 -- 172.21.15.112:0/2112534021 >> 172.21.15.112:0/2112534021 conn(0x7f04700a6410 msgr2=0x7f04700a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 -- 172.21.15.112:0/2112534021 shutdown_connections 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 -- 172.21.15.112:0/2112534021 wait complete. 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 10 monclient: build_initial_monmap 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 10 monclient: monmap: 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:45.277 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 librados: starting msgr at 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 librados: starting objecter 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 Processor -- start 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 -- start start 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 librados: setting wanted keys 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 1 librados: calling monclient init 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 10 monclient: init 2024-09-18T11:44:45.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0450003810 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.270+0000 7f046f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f0450010840 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046ed9d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f04700a0380 legacy=0x7f047014a170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046e59c700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f0450003810 0x7f04701372b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.112:54298/0 (socket says 172.21.15.112:54298) 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046dd9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0450010840 0x7f0470146a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44602/0 (socket says 172.21.15.112:44602) 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f04700a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046dd9b700 1 -- 172.21.15.112:0/959821826 learned_addr learned my addr 172.21.15.112:0/959821826 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047009efa0 con 0x7f04700a0380 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0470122a00 con 0x7f0450003810 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0470138c40 con 0x7f0450010840 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046f59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:45.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046f59e700 10 monclient(hunting): authenticate will time out at 857.116439s 2024-09-18T11:44:45.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046e59c700 1 -- 172.21.15.112:0/959821826 >> v1:172.21.15.129:6789/0 conn(0x7f0450003810 legacy=0x7f04701372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 client.?.objecter ms_handle_connect 0x7f0450003810 2024-09-18T11:44:45.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:45.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 client.?.objecter ms_handle_connect 0x7f0450010840 2024-09-18T11:44:45.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f046dd9b700 1 -- 172.21.15.112:0/959821826 >> v1:172.21.15.112:6790/0 conn(0x7f0450010840 legacy=0x7f0470146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826990341 0 0) 0x7f0470122a00 con 0x7f0450003810 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 monclient(hunting): my global_id is 4438 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0450003440 con 0x7f0450003810 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2728286034 0 0) 0x7f0450003440 con 0x7f0450003810 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0470122a00 con 0x7f0450003810 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f04640020a0 con 0x7f0450003810 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 monclient(hunting): dump: 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.282 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 955202506 0 0) 0x7f0470122a00 con 0x7f0450003810 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 >> v1:172.21.15.112:6790/0 conn(0x7f0450010840 legacy=0x7f0470146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 >> v1:172.21.15.112:6789/0 conn(0x7f04700a0380 legacy=0x7f047014a170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:45.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 1 monclient: found mon.b 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.274+0000 7f045e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0470120e00 con 0x7f0450003810 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 5 monclient: authenticate success, global_id 4438 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 10 monclient: _renew_subs 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04701223b0 con 0x7f0450003810 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 10 monclient: _renew_subs 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04701214c0 con 0x7f0450003810 2024-09-18T11:44:45.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 1 librados: init done 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0464002670 con 0x7f0450003810 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0464002980 con 0x7f0450003810 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046cd99700 4 set_mon_vals no callback set 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 10 monclient: dump: 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.286 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04540050b0 con 0x7f0450003810 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0464002be0 con 0x7f0450003810 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7f0464043740 con 0x7f0450003810 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 10 client.4438.objecter ms_dispatch 0x7f0470000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:45.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 3 client.4438.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:45.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 3 client.4438.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:45.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.278+0000 7f045e7fc700 20 client.4438.objecter dump_active .. 0 homeless 2024-09-18T11:44:45.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.282+0000 7f046dd9b700 10 monclient: get_auth_request con 0x7f0450039110 auth_method 0 2024-09-18T11:44:45.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.282+0000 7f045e7fc700 10 client.4438.objecter ms_handle_connect 0x7f0450039110 2024-09-18T11:44:45.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.290+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f04640146f0 con 0x7f0450003810 2024-09-18T11:44:45.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.290+0000 7f045e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:45.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.290+0000 7f045e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.514+0000 7f046f59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:44:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.514+0000 7f046f59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:44:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.514+0000 7f046f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.514+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f04540058e0 con 0x7f0450003810 2024-09-18T11:44:45.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.514+0000 7f045e7fc700 1 -- 172.21.15.112:0/959821826 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2519383198) 0x7f0464014510 con 0x7f0450003810 2024-09-18T11:44:45.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.514+0000 7f045e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:44:45.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.514+0000 7f045e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:45.519 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 10 librados: watch_flush enter 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 10 librados: watch_flush exit 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 20 client.4438.objecter shutdown clearing up homeless session... 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 10 client.4438.objecter successfully canceled tick 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 >> v1:172.21.15.129:6812/20806 conn(0x7f0450039110 legacy=0x7f045003b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 10 monclient: shutdown 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 >> v1:172.21.15.129:6789/0 conn(0x7f0450003810 legacy=0x7f04701372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 shutdown_connections 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 >> 172.21.15.112:0/959821826 conn(0x7f04700a6410 msgr2=0x7f04700a4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 shutdown_connections 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 1 -- 172.21.15.112:0/959821826 wait complete. 2024-09-18T11:44:45.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.518+0000 7f046f59e700 1 librados: shutdown 2024-09-18T11:44:45.533 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T11:44:45.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T11:44:45.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient: get_monmap_and_config 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient: build_initial_monmap 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient: monmap: 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:45.634176+0000 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:45.634176+0000 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:45.635 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:45.635 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 1 Processor -- start 2024-09-18T11:44:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 1 -- start start 2024-09-18T11:44:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient: init 2024-09-18T11:44:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient(hunting): picked mon.noname-b con 0x7fd2440a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient(hunting): picked mon.noname-c con 0x7fd2440a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient(hunting): picked mon.noname-a con 0x7fd2441383a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd244096f90 con 0x7fd2441383a0 2024-09-18T11:44:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440a1d90 con 0x7fd2440a0750 2024-09-18T11:44:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd244096cb0 con 0x7fd2440a0380 2024-09-18T11:44:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd248c1c700 10 monclient(hunting): authenticate will time out at 857.475534s 2024-09-18T11:44:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd243fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fd2441383a0 legacy=0x7fd24413a780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd2437fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fd2440a0380 0x7fd244097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44616/0 (socket says 172.21.15.112:44616) 2024-09-18T11:44:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.630+0000 7fd2437fe700 1 -- 172.21.15.112:0/1058121379 learned_addr learned my addr 172.21.15.112:0/1058121379 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:45.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd242ffd700 1 -- 172.21.15.112:0/1058121379 >> v1:172.21.15.129:6789/0 conn(0x7fd2440a0750 legacy=0x7fd2440a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2437fe700 1 -- 172.21.15.112:0/1058121379 >> v1:172.21.15.112:6790/0 conn(0x7fd2440a0380 legacy=0x7fd244097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616839385 0 0) 0x7fd2440a1d90 con 0x7fd2440a0750 2024-09-18T11:44:45.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient(hunting): my global_id is 4441 2024-09-18T11:44:45.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd22c003570 con 0x7fd2440a0750 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421076817 0 0) 0x7fd244096cb0 con 0x7fd2440a0380 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient(hunting): my global_id is 4451 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2440a1d90 con 0x7fd2440a0380 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764594546 0 0) 0x7fd22c003570 con 0x7fd2440a0750 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 >> v1:172.21.15.112:6790/0 conn(0x7fd2440a0380 legacy=0x7fd244097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 >> v1:172.21.15.112:6789/0 conn(0x7fd2441383a0 legacy=0x7fd24413a780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: _finish_hunting 0 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 monclient: found mon.noname-c 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd244097270 con 0x7fd2440a0750 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: _finish_auth 0 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd248c1c700 5 monclient: authenticate success, global_id 4441 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd234001990 con 0x7fd2440a0750 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: dump: 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd248c1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd234001c20 con 0x7fd2440a0750 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd248c1c700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 1 -- 172.21.15.112:0/1058121379 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd2340020a0 con 0x7fd2440a0750 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd248c1c700 4 set_mon_vals no callback set 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.634+0000 7fd2427fc700 10 monclient: dump: 2024-09-18T11:44:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient: shutdown 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- 172.21.15.112:0/1058121379 >> v1:172.21.15.129:6789/0 conn(0x7fd2440a0750 legacy=0x7fd2440a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- 172.21.15.112:0/1058121379 shutdown_connections 2024-09-18T11:44:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- 172.21.15.112:0/1058121379 >> 172.21.15.112:0/1058121379 conn(0x7fd2440a6410 msgr2=0x7fd2440a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- 172.21.15.112:0/1058121379 shutdown_connections 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- 172.21.15.112:0/1058121379 wait complete. 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient: build_initial_monmap 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient: monmap: 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 librados: starting msgr at 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 librados: starting objecter 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 Processor -- start 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- start start 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 librados: setting wanted keys 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 librados: calling monclient init 2024-09-18T11:44:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient: init 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:45.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient(hunting): picked mon.noname-c con 0x7fd2440a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient(hunting): picked mon.noname-b con 0x7fd2440a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient(hunting): picked mon.noname-a con 0x7fd2441383a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24409ac00 con 0x7fd2441383a0 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd244099370 con 0x7fd2440a0750 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440a2070 con 0x7fd2440a0380 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd248c1c700 10 monclient(hunting): authenticate will time out at 857.482103s 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd243fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fd2441383a0 legacy=0x7fd244149ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd2437fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fd2440a0380 0x7fd24409e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44626/0 (socket says 172.21.15.112:44626) 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.638+0000 7fd2437fe700 1 -- 172.21.15.112:0/4176199961 learned_addr learned my addr 172.21.15.112:0/4176199961 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:45.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd242ffd700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.129:6789/0 conn(0x7fd2440a0750 legacy=0x7fd244146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 client.?.objecter ms_handle_connect 0x7fd2440a0750 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 client.?.objecter ms_handle_connect 0x7fd2440a0380 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd2437fe700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6790/0 conn(0x7fd2440a0380 legacy=0x7fd24409e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163195485 0 0) 0x7fd244099370 con 0x7fd2440a0750 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): my global_id is 4444 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230004f10 con 0x7fd2440a0750 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272439744 0 0) 0x7fd2440a2070 con 0x7fd2440a0380 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): my global_id is 4454 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd244099370 con 0x7fd2440a0380 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1340351290 0 0) 0x7fd244099370 con 0x7fd2440a0380 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2440a2070 con 0x7fd2440a0380 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd238001c50 con 0x7fd2440a0380 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): dump: 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.656 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 259345504 0 0) 0x7fd230004f10 con 0x7fd2440a0750 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd244099370 con 0x7fd2440a0750 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd234001f00 con 0x7fd2440a0750 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient(hunting): dump: 2024-09-18T11:44:45.657 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3744353558 0 0) 0x7fd2440a2070 con 0x7fd2440a0380 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.129:6789/0 conn(0x7fd2440a0750 legacy=0x7fd244146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6789/0 conn(0x7fd2441383a0 legacy=0x7fd244149ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:44:45.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 monclient: found mon.c 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd244099a30 con 0x7fd2440a0380 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient: _finish_auth 0 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 5 monclient: authenticate success, global_id 4454 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 10 monclient: _renew_subs 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24409a920 con 0x7fd2440a0380 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:45.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 10 monclient: _renew_subs 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd244141930 con 0x7fd2440a0380 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 1 librados: init done 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd238004ae0 con 0x7fd2440a0380 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd238004e70 con 0x7fd2440a0380 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd2417fa700 4 set_mon_vals no callback set 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 monclient: dump: 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.660 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fd238005050 con 0x7fd2440a0380 2024-09-18T11:44:45.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd224006590 con 0x7fd224002840 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fd2380437c0 con 0x7fd2440a0380 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 10 client.4454.objecter ms_dispatch 0x7fd244000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 3 client.4454.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 3 client.4454.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.642+0000 7fd22b7fe700 20 client.4454.objecter dump_active .. 0 homeless 2024-09-18T11:44:45.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd243fff700 10 monclient: get_auth_request con 0x7fd230038260 auth_method 0 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 client.4454.objecter ms_handle_connect 0x7fd224002840 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 client.4454.objecter resend_mon_ops 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd242ffd700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6790/0 conn(0x7fd224002840 legacy=0x7fd224004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455779623 0 0) 0x7fd224006590 con 0x7fd224002840 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 monclient(hunting): my global_id is 4457 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2440a2070 con 0x7fd224002840 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3834191464 0 0) 0x7fd2440a2070 con 0x7fd224002840 2024-09-18T11:44:45.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd224006590 con 0x7fd224002840 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd2340025d0 con 0x7fd224002840 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 monclient: dump: 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.664 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3208342875 0 0) 0x7fd224006590 con 0x7fd224002840 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd22400a140 con 0x7fd224002840 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 client.4454.objecter ms_handle_connect 0x7fd230038260 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fd2340027c0 con 0x7fd224002840 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6790/0 conn(0x7fd224002840 legacy=0x7fd224004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.646+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6790/0 conn(0x7fd224002840 legacy=0x7fd224004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:45.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.658+0000 7fd248c1c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.658+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd224009670 con 0x7fd2240090e0 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.658+0000 7fd242ffd700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6790/0 conn(0x7fd2240090e0 legacy=0x7fd224008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.658+0000 7fd22b7fe700 10 client.4454.objecter ms_handle_connect 0x7fd2240090e0 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.658+0000 7fd22b7fe700 10 client.4454.objecter resend_mon_ops 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.658+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450027498 0 0) 0x7fd224009670 con 0x7fd2240090e0 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 10 monclient(hunting): my global_id is 4460 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd224006590 con 0x7fd2240090e0 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 852994327 0 0) 0x7fd224006590 con 0x7fd2240090e0 2024-09-18T11:44:45.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd224009670 con 0x7fd2240090e0 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd234005810 con 0x7fd2240090e0 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 10 monclient: dump: 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.669 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1084050172 0 0) 0x7fd224009670 con 0x7fd2240090e0 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd224009930 con 0x7fd2240090e0 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3820 (unknown 0 0 1620246730) 0x7fd2340079b0 con 0x7fd2240090e0 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6790/0 conn(0x7fd2240090e0 legacy=0x7fd224008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd22b7fe700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6790/0 conn(0x7fd2240090e0 legacy=0x7fd224008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 10, 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:44:45.671 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:45.672 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:45.673 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:45.674 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:45.675 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.676 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:44:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:44:45.682 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 10 librados: watch_flush enter 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 10 librados: watch_flush exit 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 20 client.4454.objecter shutdown clearing up homeless session... 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 10 client.4454.objecter successfully canceled tick 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.129:6812/20806 conn(0x7fd230038260 legacy=0x7fd23003a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 10 monclient: shutdown 2024-09-18T11:44:45.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 >> v1:172.21.15.112:6790/0 conn(0x7fd2440a0380 legacy=0x7fd24409e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd243fff700 1 -- 172.21.15.112:0/4176199961 reap_dead start 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd243fff700 1 -- 172.21.15.112:0/4176199961 reap_dead start 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 shutdown_connections 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 >> 172.21.15.112:0/4176199961 conn(0x7fd2440a6410 msgr2=0x7fd2441375c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 shutdown_connections 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 1 -- 172.21.15.112:0/4176199961 wait complete. 2024-09-18T11:44:45.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.662+0000 7fd248c1c700 1 librados: shutdown 2024-09-18T11:44:45.685 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:44:45.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T11:44:45.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:44:45.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.790+0000 7fb24f7f7700 10 monclient: get_monmap_and_config 2024-09-18T11:44:45.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.790+0000 7fb24f7f7700 10 monclient: build_initial_monmap 2024-09-18T11:44:45.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.790+0000 7fb24f7f7700 10 monclient: monmap: 2024-09-18T11:44:45.795 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:45.795 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.796 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:44:45.795147+0000 2024-09-18T11:44:45.796 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:44:45.795147+0000 2024-09-18T11:44:45.796 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:45.796 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.796 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:45.796 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:44:45.797 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:44:45.797 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.790+0000 7fb24f7f7700 1 Processor -- start 2024-09-18T11:44:45.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 1 -- start start 2024-09-18T11:44:45.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 10 monclient: init 2024-09-18T11:44:45.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:45.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:44:45.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 10 monclient(hunting): picked mon.noname-a con 0x7fb2480a2270 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:45.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 10 monclient(hunting): picked mon.noname-b con 0x7fb24806aa70 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:45.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 10 monclient(hunting): picked mon.noname-c con 0x7fb24806cb80 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:45.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24806a830 con 0x7fb2480a2270 2024-09-18T11:44:45.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2480a2640 con 0x7fb24806cb80 2024-09-18T11:44:45.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24806d000 con 0x7fb24806aa70 2024-09-18T11:44:45.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:45.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24f7f7700 10 monclient(hunting): authenticate will time out at 857.636913s 2024-09-18T11:44:45.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24d593700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fb2480a2270 legacy=0x7fb2480a0070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:45.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24cd92700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb24806aa70 0x7fb2480a0780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44662/0 (socket says 172.21.15.112:44662) 2024-09-18T11:44:45.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24cd92700 1 -- 172.21.15.112:0/2155438675 learned_addr learned my addr 172.21.15.112:0/2155438675 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:45.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24dd94700 1 -- 172.21.15.112:0/2155438675 >> v1:172.21.15.129:6789/0 conn(0x7fb24806cb80 legacy=0x7fb248137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb24cd92700 1 -- 172.21.15.112:0/2155438675 >> v1:172.21.15.112:6790/0 conn(0x7fb24806aa70 legacy=0x7fb2480a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230997347 0 0) 0x7fb24806d000 con 0x7fb24806aa70 2024-09-18T11:44:45.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 10 monclient(hunting): my global_id is 4463 2024-09-18T11:44:45.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb234004680 con 0x7fb24806aa70 2024-09-18T11:44:45.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112412024 0 0) 0x7fb2480a2640 con 0x7fb24806cb80 2024-09-18T11:44:45.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 10 monclient(hunting): my global_id is 4447 2024-09-18T11:44:45.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.794+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24806d000 con 0x7fb24806cb80 2024-09-18T11:44:45.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2319770804 0 0) 0x7fb234004680 con 0x7fb24806aa70 2024-09-18T11:44:45.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 >> v1:172.21.15.129:6789/0 conn(0x7fb24806cb80 legacy=0x7fb248137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 >> v1:172.21.15.112:6789/0 conn(0x7fb2480a2270 legacy=0x7fb2480a0070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:44:45.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: _finish_hunting 0 2024-09-18T11:44:45.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 1 monclient: found mon.noname-b 2024-09-18T11:44:45.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:45.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:44:45.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2480a0e90 con 0x7fb24806aa70 2024-09-18T11:44:45.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: _finish_auth 0 2024-09-18T11:44:45.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:45.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb23c001940 con 0x7fb24806aa70 2024-09-18T11:44:45.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 5 monclient: authenticate success, global_id 4463 2024-09-18T11:44:45.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:45.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:45.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: dump: 2024-09-18T11:44:45.825 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.825 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.825 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.825 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.826 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.826 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.826 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.826 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.826 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.826 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:44:45.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb23c001bd0 con 0x7fb24806aa70 2024-09-18T11:44:45.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:45.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 1 -- 172.21.15.112:0/2155438675 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb23c004a90 con 0x7fb24806aa70 2024-09-18T11:44:45.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 10 monclient: get_monmap_and_config success 2024-09-18T11:44:45.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 4 set_mon_vals no callback set 2024-09-18T11:44:45.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:45.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb23bfff700 10 monclient: dump: 2024-09-18T11:44:45.828 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.828 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.829 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.829 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.829 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.829 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.829 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.829 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.829 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.829 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 10 monclient: shutdown 2024-09-18T11:44:45.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:45.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 -- 172.21.15.112:0/2155438675 >> v1:172.21.15.112:6790/0 conn(0x7fb24806aa70 legacy=0x7fb2480a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 -- 172.21.15.112:0/2155438675 shutdown_connections 2024-09-18T11:44:45.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 -- 172.21.15.112:0/2155438675 >> 172.21.15.112:0/2155438675 conn(0x7fb2480680d0 msgr2=0x7fb2480a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:45.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 -- 172.21.15.112:0/2155438675 shutdown_connections 2024-09-18T11:44:45.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 -- 172.21.15.112:0/2155438675 wait complete. 2024-09-18T11:44:45.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 10 monclient: build_initial_monmap 2024-09-18T11:44:45.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 10 monclient: monmap: 2024-09-18T11:44:45.831 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:44:45.831 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:44:45.831 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:44:45.831 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:44:45.831 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:44:45.832 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.832 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:44:45.832 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:44:45.832 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:44:45.832 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 librados: starting msgr at 2024-09-18T11:44:45.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 librados: starting objecter 2024-09-18T11:44:45.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 Processor -- start 2024-09-18T11:44:45.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 -- start start 2024-09-18T11:44:45.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 librados: setting wanted keys 2024-09-18T11:44:45.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 1 librados: calling monclient init 2024-09-18T11:44:45.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.798+0000 7fb24f7f7700 10 monclient: init 2024-09-18T11:44:45.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 10 monclient: _reopen_session rank -1 2024-09-18T11:44:45.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:44:45.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 10 monclient(hunting): picked mon.noname-c con 0x7fb24806aa70 addr v1:172.21.15.112:6790/0 2024-09-18T11:44:45.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 10 monclient(hunting): picked mon.noname-b con 0x7fb24806cb80 addr v1:172.21.15.129:6789/0 2024-09-18T11:44:45.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 10 monclient(hunting): picked mon.noname-a con 0x7fb2480a2270 addr v1:172.21.15.112:6789/0 2024-09-18T11:44:45.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24806b2c0 con 0x7fb2480a2270 2024-09-18T11:44:45.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24809d560 con 0x7fb24806cb80 2024-09-18T11:44:45.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24811eec0 con 0x7fb24806aa70 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24dd94700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fb2480a2270 legacy=0x7fb248147880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 10 monclient(hunting): _renew_subs 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24f7f7700 10 monclient(hunting): authenticate will time out at 857.644459s 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24d593700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb24806aa70 0x7fb24813f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44664/0 (socket says 172.21.15.112:44664) 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24d593700 1 -- 172.21.15.112:0/643360106 learned_addr learned my addr 172.21.15.112:0/643360106 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 client.?.objecter ms_handle_connect 0x7fb24806cb80 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24cd92700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6789/0 conn(0x7fb24806cb80 legacy=0x7fb2481333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 client.?.objecter ms_handle_connect 0x7fb24806aa70 2024-09-18T11:44:45.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:44:45.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb24d593700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.112:6790/0 conn(0x7fb24806aa70 legacy=0x7fb24813f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459599663 0 0) 0x7fb24809d560 con 0x7fb24806cb80 2024-09-18T11:44:45.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): my global_id is 4450 2024-09-18T11:44:45.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb22c004f10 con 0x7fb24806cb80 2024-09-18T11:44:45.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756496884 0 0) 0x7fb24811eec0 con 0x7fb24806aa70 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): my global_id is 4466 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24809d560 con 0x7fb24806aa70 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2013093492 0 0) 0x7fb22c004f10 con 0x7fb24806cb80 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb24811eec0 con 0x7fb24806cb80 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb23c004830 con 0x7fb24806cb80 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:44:45.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): dump: 2024-09-18T11:44:45.838 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.838 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.838 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.838 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.838 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.839 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.839 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.839 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.839 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.839 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:44:45.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387380715 0 0) 0x7fb24809d560 con 0x7fb24806aa70 2024-09-18T11:44:45.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb22c004f10 con 0x7fb24806aa70 2024-09-18T11:44:45.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb244004520 con 0x7fb24806aa70 2024-09-18T11:44:45.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:44:45.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient(hunting): dump: 2024-09-18T11:44:45.841 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.841 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.841 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.841 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.841 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.841 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.842 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.842 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.842 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.842 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1336125455 0 0) 0x7fb24811eec0 con 0x7fb24806cb80 2024-09-18T11:44:45.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.112:6790/0 conn(0x7fb24806aa70 legacy=0x7fb24813f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.112:6789/0 conn(0x7fb2480a2270 legacy=0x7fb248147880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:44:45.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient: _finish_hunting 0 2024-09-18T11:44:45.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 monclient: found mon.b 2024-09-18T11:44:45.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:44:45.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24811e9b0 con 0x7fb24806cb80 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 10 monclient: _finish_auth 0 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.802+0000 7fb2397fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 5 monclient: authenticate success, global_id 4450 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 10 monclient: _renew_subs 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb248140360 con 0x7fb24806cb80 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 10 monclient: _renew_subs 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:44:45.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb248133ad0 con 0x7fb24806cb80 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 1 librados: init done 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb23c001580 con 0x7fb24806cb80 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb23c004f30 con 0x7fb24806cb80 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb23b7fe700 4 set_mon_vals no callback set 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 10 monclient: dump: 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.845 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.846 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.846 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.846 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.846 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.846 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.846 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.846 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:45.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb228006590 con 0x7fb228002840 2024-09-18T11:44:45.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb23c00f8b0 con 0x7fb24806cb80 2024-09-18T11:44:45.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fb23c0437b0 con 0x7fb24806cb80 2024-09-18T11:44:45.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 10 client.4450.objecter ms_dispatch 0x7fb248000fd0 osd_map(18..18 src has 1..18) v4 2024-09-18T11:44:45.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 3 client.4450.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:44:45.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 3 client.4450.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:44:45.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb2397fa700 20 client.4450.objecter dump_active .. 0 homeless 2024-09-18T11:44:45.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.806+0000 7fb24dd94700 10 monclient: get_auth_request con 0x7fb22c0382b0 auth_method 0 2024-09-18T11:44:45.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 client.4450.objecter ms_handle_connect 0x7fb228002840 2024-09-18T11:44:45.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb24d593700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6789/0 conn(0x7fb228002840 legacy=0x7fb228004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 client.4450.objecter resend_mon_ops 2024-09-18T11:44:45.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161456031 0 0) 0x7fb228006590 con 0x7fb228002840 2024-09-18T11:44:45.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 monclient(hunting): my global_id is 4453 2024-09-18T11:44:45.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24811eec0 con 0x7fb228002840 2024-09-18T11:44:45.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2280198180 0 0) 0x7fb24811eec0 con 0x7fb228002840 2024-09-18T11:44:45.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb228006590 con 0x7fb228002840 2024-09-18T11:44:45.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 client.4450.objecter ms_handle_connect 0x7fb22c0382b0 2024-09-18T11:44:45.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb244004b40 con 0x7fb228002840 2024-09-18T11:44:45.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:45.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 monclient: dump: 2024-09-18T11:44:45.850 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.850 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.852 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1015306137 0 0) 0x7fb228006590 con 0x7fb228002840 2024-09-18T11:44:45.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb22800a140 con 0x7fb228002840 2024-09-18T11:44:45.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fb244004f10 con 0x7fb228002840 2024-09-18T11:44:45.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:44:45.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:44:45.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6789/0 conn(0x7fb228002840 legacy=0x7fb228004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.810+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6789/0 conn(0x7fb228002840 legacy=0x7fb228004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:45.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb24f7f7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:44:45.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb228009f10 con 0x7fb228006900 2024-09-18T11:44:45.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb24d593700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6789/0 conn(0x7fb228006900 legacy=0x7fb228006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:44:45.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb2397fa700 10 client.4450.objecter ms_handle_connect 0x7fb228006900 2024-09-18T11:44:45.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb2397fa700 10 client.4450.objecter resend_mon_ops 2024-09-18T11:44:45.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824412984 0 0) 0x7fb228009f10 con 0x7fb228006900 2024-09-18T11:44:45.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb2397fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:44:45.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb2397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:44:45.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb2397fa700 10 monclient(hunting): my global_id is 4456 2024-09-18T11:44:45.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.822+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb228006590 con 0x7fb228006900 2024-09-18T11:44:45.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4001899978 0 0) 0x7fb228006590 con 0x7fb228006900 2024-09-18T11:44:45.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb228009f10 con 0x7fb228006900 2024-09-18T11:44:45.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb2440054d0 con 0x7fb228006900 2024-09-18T11:44:45.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:44:45.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:44:45.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 10 monclient: dump: 2024-09-18T11:44:45.858 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:44:45.858 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:44:45.858 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.858 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:44:45.859 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:44:45.859 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:44:45.859 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:44:45.859 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:44:45.859 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:44:45.859 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:44:45.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1134740981 0 0) 0x7fb228009f10 con 0x7fb228006900 2024-09-18T11:44:45.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb228008760 con 0x7fb228006900 2024-09-18T11:44:45.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4162 (unknown 0 0 1467831599) 0x7fb244006910 con 0x7fb228006900 2024-09-18T11:44:45.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:44:45.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:44:45.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6789/0 conn(0x7fb228006900 legacy=0x7fb228006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb2397fa700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6789/0 conn(0x7fb228006900 legacy=0x7fb228006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:44:45.861 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:44:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:44:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:44:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:44:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 10, 2024-09-18T11:44:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:44:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:44:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:44:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:44:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:44:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:44:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:44:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:44:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:44:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:44:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:44:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:44:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 6 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:44:45.874 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:44:45.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 10 librados: watch_flush enter 2024-09-18T11:44:45.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 10 librados: watch_flush exit 2024-09-18T11:44:45.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 20 client.4450.objecter shutdown clearing up homeless session... 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 10 client.4450.objecter successfully canceled tick 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6812/20806 conn(0x7fb22c0382b0 legacy=0x7fb22c03a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 10 monclient: shutdown 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 >> v1:172.21.15.129:6789/0 conn(0x7fb24806cb80 legacy=0x7fb2481333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24dd94700 1 -- 172.21.15.112:0/643360106 reap_dead start 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24dd94700 1 -- 172.21.15.112:0/643360106 reap_dead start 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 shutdown_connections 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 >> 172.21.15.112:0/643360106 conn(0x7fb2480680d0 msgr2=0x7fb24809d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 shutdown_connections 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 1 -- 172.21.15.112:0/643360106 wait complete. 2024-09-18T11:44:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:44:45.826+0000 7fb24f7f7700 1 librados: shutdown 2024-09-18T11:44:45.878 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:44:45.878 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T11:45:22.676 INFO:tasks.workunit.client.0.smithi129.stderr:Note: switching to '7061ea6ca50908d104e9f3979dc615e118d9e153'. 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr: git switch -c 2024-09-18T11:45:22.677 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-18T11:45:22.678 INFO:tasks.workunit.client.0.smithi129.stderr:Or undo this operation with: 2024-09-18T11:45:22.678 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-18T11:45:22.678 INFO:tasks.workunit.client.0.smithi129.stderr: git switch - 2024-09-18T11:45:22.678 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-18T11:45:22.678 INFO:tasks.workunit.client.0.smithi129.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T11:45:22.678 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-09-18T11:45:22.678 INFO:tasks.workunit.client.0.smithi129.stderr:HEAD is now at 7061ea6ca50 Merge branch 'wip-65946-quincy' of https://github.com/amathuria/ceph into wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:45:22.682 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-18T11:45:22.745 INFO:tasks.workunit.client.0.smithi129.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-18T11:45:22.748 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-18T11:45:22.749 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-18T11:45:23.571 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-18T11:45:23.678 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-18T11:45:23.740 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-18T11:45:23.742 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-18T11:45:23.742 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-18T11:45:23.803 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-18T11:45:23.808 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-18T11:45:23.808 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-18T11:45:23.814 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-18T11:45:23.815 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-18T11:45:23.815 DEBUG:teuthology.orchestra.run.smithi129:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-18T11:45:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ parallel=1 2024-09-18T11:45:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' '' = --serial ']' 2024-09-18T11:45:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ color= 2024-09-18T11:45:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -t 1 ']' 2024-09-18T11:45:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-18T11:45:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ declare -A pids 2024-09-18T11:45:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_aio 2024-09-18T11:45:23.830 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_aio' 2024-09-18T11:45:23.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_aio 2024-09-18T11:45:23.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.834 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_aio 2024-09-18T11:45:23.834 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23760 2024-09-18T11:45:23.834 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_aio on pid 23760' 2024-09-18T11:45:23.834 INFO:tasks.workunit.client.0.smithi129.stdout:test api_aio on pid 23760 2024-09-18T11:45:23.835 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23760 2024-09-18T11:45:23.835 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.835 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.835 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-18T11:45:23.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_aio_pp 2024-09-18T11:45:23.836 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_aio_pp' 2024-09-18T11:45:23.836 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_aio_pp 2024-09-18T11:45:23.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.837 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.837 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.837 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.837 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.837 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_aio 2024-09-18T11:45:23.837 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-18T11:45:23.837 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_aio: /' 2024-09-18T11:45:23.840 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_aio_pp 2024-09-18T11:45:23.840 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23768 2024-09-18T11:45:23.840 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_aio_pp on pid 23768' 2024-09-18T11:45:23.841 INFO:tasks.workunit.client.0.smithi129.stdout:test api_aio_pp on pid 23768 2024-09-18T11:45:23.843 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23768 2024-09-18T11:45:23.843 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.843 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.843 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-18T11:45:23.843 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_io 2024-09-18T11:45:23.843 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_io' 2024-09-18T11:45:23.844 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_io 2024-09-18T11:45:23.844 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.844 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.844 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.844 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.844 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.844 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_aio_pp 2024-09-18T11:45:23.845 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-18T11:45:23.846 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-18T11:45:23.846 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_io 2024-09-18T11:45:23.846 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23776 2024-09-18T11:45:23.847 INFO:tasks.workunit.client.0.smithi129.stdout:test api_io on pid 23776 2024-09-18T11:45:23.849 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_io on pid 23776' 2024-09-18T11:45:23.849 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-18T11:45:23.849 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23776 2024-09-18T11:45:23.849 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.849 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_io_pp 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_io.log 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_io 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_io: /' 2024-09-18T11:45:23.850 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_io_pp' 2024-09-18T11:45:23.851 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_io_pp 2024-09-18T11:45:23.851 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_io_pp 2024-09-18T11:45:23.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23784 2024-09-18T11:45:23.853 INFO:tasks.workunit.client.0.smithi129.stdout:test api_io_pp on pid 23784 2024-09-18T11:45:23.854 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_io_pp on pid 23784' 2024-09-18T11:45:23.854 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23784 2024-09-18T11:45:23.854 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-18T11:45:23.854 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.854 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_asio 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_asio' 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_asio 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_io_pp 2024-09-18T11:45:23.855 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-18T11:45:23.857 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_asio 2024-09-18T11:45:23.857 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23792 2024-09-18T11:45:23.857 INFO:tasks.workunit.client.0.smithi129.stdout:test api_asio on pid 23792 2024-09-18T11:45:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_asio on pid 23792' 2024-09-18T11:45:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23792 2024-09-18T11:45:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-18T11:45:23.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.860 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.860 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.860 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.860 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.860 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.862 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_asio: /' 2024-09-18T11:45:23.862 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_asio 2024-09-18T11:45:23.863 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-18T11:45:23.863 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_list 2024-09-18T11:45:23.863 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_list' 2024-09-18T11:45:23.863 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_list 2024-09-18T11:45:23.863 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_list 2024-09-18T11:45:23.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23800 2024-09-18T11:45:23.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_list on pid 23800' 2024-09-18T11:45:23.866 INFO:tasks.workunit.client.0.smithi129.stdout:test api_list on pid 23800 2024-09-18T11:45:23.869 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23800 2024-09-18T11:45:23.870 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.870 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.870 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-18T11:45:23.870 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_lock 2024-09-18T11:45:23.870 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_lock' 2024-09-18T11:45:23.870 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_lock 2024-09-18T11:45:23.871 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.872 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_list: /' 2024-09-18T11:45:23.872 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_list 2024-09-18T11:45:23.872 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_list.log 2024-09-18T11:45:23.873 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_lock 2024-09-18T11:45:23.873 INFO:tasks.workunit.client.0.smithi129.stdout:test api_lock on pid 23815 2024-09-18T11:45:23.874 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23815 2024-09-18T11:45:23.874 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_lock on pid 23815' 2024-09-18T11:45:23.875 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23815 2024-09-18T11:45:23.875 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.875 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.875 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_lock_pp 2024-09-18T11:45:23.875 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_lock_pp' 2024-09-18T11:45:23.876 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-18T11:45:23.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.878 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.878 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.878 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-18T11:45:23.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_lock_pp 2024-09-18T11:45:23.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.879 INFO:tasks.workunit.client.0.smithi129.stdout:test api_lock_pp on pid 23831 2024-09-18T11:45:23.880 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_lock_pp 2024-09-18T11:45:23.880 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23831 2024-09-18T11:45:23.881 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_lock_pp on pid 23831' 2024-09-18T11:45:23.881 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23831 2024-09-18T11:45:23.881 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.881 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.881 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_misc 2024-09-18T11:45:23.882 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_misc' 2024-09-18T11:45:23.882 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_lock 2024-09-18T11:45:23.882 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_lock: /' 2024-09-18T11:45:23.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_misc 2024-09-18T11:45:23.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.885 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-18T11:45:23.888 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_lock_pp 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_misc 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23862 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_misc on pid 23862' 2024-09-18T11:45:23.889 INFO:tasks.workunit.client.0.smithi129.stdout:test api_misc on pid 23862 2024-09-18T11:45:23.890 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23862 2024-09-18T11:45:23.890 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.891 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-18T11:45:23.891 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.891 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.891 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.891 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.891 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.891 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_misc_pp 2024-09-18T11:45:23.891 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_misc_pp' 2024-09-18T11:45:23.892 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_misc 2024-09-18T11:45:23.892 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-18T11:45:23.892 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_misc: /' 2024-09-18T11:45:23.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_misc_pp 2024-09-18T11:45:23.905 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-18T11:45:23.905 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] Global test environment set-up. 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient: get_monmap_and_config 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient: build_initial_monmap 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient: monmap: 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 0 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 0.000000 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 0.000000 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:45:23.906 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 1 Processor -- start 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 1 -- start start 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient: init 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient: _reopen_session rank -1 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient(hunting): picked mon.noname-c con 0x5556dc894430 addr v1:172.21.15.112:6790/0 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient(hunting): picked mon.noname-a con 0x5556dc896480 addr v1:172.21.15.112:6789/0 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient(hunting): picked mon.noname-b con 0x5556dc92f3a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5556dc7c61d0 con 0x5556dc896480 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5556dc8670f0 con 0x5556dc92f3a0 2024-09-18T11:45:23.907 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5556dc87d910 con 0x5556dc894430 2024-09-18T11:45:23.908 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient(hunting): _renew_subs 2024-09-18T11:45:23.908 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c61836c0 10 monclient(hunting): authenticate will time out at 897.551749s 2024-09-18T11:45:23.908 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.888+0000 7fd8c34bb700 1 -- >> v1:172.21.15.112:6789/0 conn(0x5556dc896480 legacy=0x5556dc88e5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:45:23.908 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.892+0000 7fd8c44bd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x5556dc92f3a0 0x5556dc942a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:37906/0 (socket says 172.21.15.129:37906) 2024-09-18T11:45:23.908 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.892+0000 7fd8c44bd700 1 -- 172.21.15.129:0/2949574578 learned_addr learned my addr 172.21.15.129:0/2949574578 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:45:23.908 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.892+0000 7fd8c44bd700 1 -- 172.21.15.129:0/2949574578 >> v1:172.21.15.129:6789/0 conn(0x5556dc92f3a0 legacy=0x5556dc942a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:45:23.908 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.892+0000 7fd8c3cbc700 1 -- 172.21.15.129:0/2949574578 >> v1:172.21.15.112:6790/0 conn(0x5556dc894430 legacy=0x5556dc88dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:45:23.908 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047797469 0 0) 0x5556dc87d910 con 0x5556dc894430 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient(hunting): my global_id is 4490 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b0001e10 con 0x5556dc894430 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114736414 0 0) 0x7fd8b0001e10 con 0x5556dc894430 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 >> v1:172.21.15.129:6789/0 conn(0x5556dc92f3a0 legacy=0x5556dc942a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 >> v1:172.21.15.112:6789/0 conn(0x5556dc896480 legacy=0x5556dc88e5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: _finish_hunting 0 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 monclient: found mon.noname-c 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.112:6790/0 2024-09-18T11:45:23.909 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556dc87d710 con 0x5556dc894430 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: _finish_auth 0 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd8b4002b50 con 0x5556dc894430 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: dump: 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 1 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2024-09-18T11:45:23.910 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:45:23.911 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:45:23.911 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:45:23.911 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.911 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c61836c0 5 monclient: authenticate success, global_id 4490 2024-09-18T11:45:23.911 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c61836c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:45:23.911 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b4002da0 con 0x5556dc894430 2024-09-18T11:45:23.911 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:45:23.912 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 1 -- 172.21.15.129:0/2949574578 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd8b4003130 con 0x5556dc894430 2024-09-18T11:45:23.912 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:45:23.912 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:45:23.912 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c24b9700 10 monclient: dump: 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 1 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c61836c0 10 monclient: get_monmap_and_config success 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c61836c0 4 set_mon_vals no callback set 2024-09-18T11:45:23.913 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c61836c0 10 monclient: shutdown 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c61836c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.900+0000 7fd8c61836c0 1 -- 172.21.15.129:0/2949574578 >> v1:172.21.15.112:6790/0 conn(0x5556dc894430 legacy=0x5556dc88dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 -- 172.21.15.129:0/2949574578 shutdown_connections 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 -- 172.21.15.129:0/2949574578 >> 172.21.15.129:0/2949574578 conn(0x5556dc899690 msgr2=0x5556dc87bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 -- 172.21.15.129:0/2949574578 shutdown_connections 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 -- 172.21.15.129:0/2949574578 wait complete. 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient: build_initial_monmap 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient: monmap: 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 0 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 0.000000 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 0.000000 2024-09-18T11:45:23.914 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 librados: starting msgr at 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 librados: starting objecter 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 Processor -- start 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 -- start start 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 librados: setting wanted keys 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 librados: calling monclient init 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient: init 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient: _reopen_session rank -1 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:45:23.916 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_misc_pp 2024-09-18T11:45:23.916 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23933 2024-09-18T11:45:23.916 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_misc_pp on pid 23933' 2024-09-18T11:45:23.917 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient(hunting): picked mon.noname-c con 0x5556dc894430test api_misc_pp on pid 23933 2024-09-18T11:45:23.917 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23933 2024-09-18T11:45:23.917 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.917 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: addr v1:172.21.15.112:6790/0 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient(hunting): picked mon.noname-a con 0x5556dc896480 addr v1:172.21.15.112:6789/0 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient(hunting): picked mon.noname-b con 0x5556dc92f3a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5556dc86e790 con 0x5556dc896480 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5556dc86e530 con 0x5556dc92f3a0 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5556dc8930b0 con 0x5556dc894430 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient(hunting): _renew_subs 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c61836c0 10 monclient(hunting): authenticate will time out at 897.567130s 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c34bb700 1 -- >> v1:172.21.15.112:6789/0 conn(0x5556dc896480 legacy=0x5556dc86de20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:45:23.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c44bd700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x5556dc92f3a0 0x5556dc952870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:37938/0 (socket says 172.21.15.129:37938) 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 learned_addr learned my addr 172.21.15.129:0/1652477188 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.904+0000 7fd8c3cbc700 1 --1- 172.21.15.129:0/1652477188 >> v1:172.21.15.112:6790/0 conn(0x5556dc894430 0x5556dc86d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.129:33914/0 (socket says 172.21.15.129:33914) 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 >> v1:172.21.15.129:6789/0 conn(0x5556dc92f3a0 legacy=0x5556dc952870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 client.?.objecter ms_handle_connect 0x5556dc92f3a0 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 client.?.objecter resend_mon_ops 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c3cbc700 1 -- 172.21.15.129:0/1652477188 >> v1:172.21.15.112:6790/0 conn(0x5556dc894430 legacy=0x5556dc86d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 client.?.objecter ms_handle_connect 0x5556dc894430 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 client.?.objecter resend_mon_ops 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265798893 0 0) 0x5556dc8930b0 con 0x5556dc894430 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient(hunting): my global_id is 4502 2024-09-18T11:45:23.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b0002050 con 0x5556dc894430 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1949186549 0 0) 0x7fd8b0002050 con 0x5556dc894430 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5556dc8930b0 con 0x5556dc894430 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd8b4003170 con 0x5556dc894430 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient(hunting): dump: 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 1 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2024-09-18T11:45:23.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4066561748 0 0) 0x5556dc8930b0 con 0x5556dc894430 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 >> v1:172.21.15.129:6789/0 conn(0x5556dc92f3a0 legacy=0x5556dc952870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 >> v1:172.21.15.112:6789/0 conn(0x5556dc896480 legacy=0x5556dc86de20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient: _finish_hunting 0 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 monclient: found mon.c 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:45:23.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556dc92f0d0 con 0x5556dc894430 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient: _finish_auth 0 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 5 monclient: authenticate success, global_id 4502 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 10 monclient: _renew_subs 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x5556dc875c90 con 0x5556dc894430 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 10 monclient: _renew_subs 2024-09-18T11:45:23.922 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x5556dc86e960 con 0x5556dc894430 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 1 librados: init done 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c61836c0 10 librados: wait_for_osdmap waiting 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b4003770 con 0x5556dc894430 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fd8b4003ac0 con 0x5556dc894430 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:45:23.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c0cb6700 10 monclient: dump: 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: epoch 1 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: election_strategy: 1 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.908+0000 7fd8c24b9700 4 set_mon_vals no callback set 2024-09-18T11:45:23.924 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fd8b4003c20 con 0x5556dc894430 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3530+0+0 (unknown 2490895846 0 0) 0x7fd8b40422f0 con 0x5556dc894430 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_map(18..18 src has 1..18) v4 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 3 client.4502.objecter handle_osd_map got epochs [18,18] > 0 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 3 client.4502.objecter handle_osd_map decoding full epoch 18 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c34bb700 10 monclient: get_auth_request con 0x7fd8b00362c0 auth_method 0 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 20 client.4502.objecter dump_active .. 0 homeless 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c61836c0 10 librados: wait_for_osdmap done waiting 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c61836c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.112:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8b8001c60 con 0x5556dc894430 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (unknown 77724187 0 0) 0x7fd8b40136c0 con 0x5556dc894430 2024-09-18T11:45:23.925 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-18T11:45:23.926 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_tier_pp 2024-09-18T11:45:23.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_tier_pp' 2024-09-18T11:45:23.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_misc_pp 2024-09-18T11:45:23.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-18T11:45:23.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-18T11:45:23.928 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_tier_pp 2024-09-18T11:45:23.928 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_tier_pp 2024-09-18T11:45:23.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23961 2024-09-18T11:45:23.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_tier_pp on pid 23961' 2024-09-18T11:45:23.932 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c1cb8700 test api_tier_pp on pid 23961 2024-09-18T11:45:23.932 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23961 2024-09-18T11:45:23.932 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.932 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.933 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_pool 2024-09-18T11:45:23.933 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-18T11:45:23.933 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_pool' 2024-09-18T11:45:23.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_pool 2024-09-18T11:45:23.934 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.934 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.934 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.934 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.935 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-18T11:45:23.935 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_tier_pp 2024-09-18T11:45:23.935 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-18T11:45:23.937 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_pool 2024-09-18T11:45:23.937 INFO:tasks.workunit.client.0.smithi129.stdout:test api_pool on pid 23982 2024-09-18T11:45:23.937 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23982 2024-09-18T11:45:23.937 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_pool on pid 23982' 2024-09-18T11:45:23.938 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=23982 2024-09-18T11:45:23.938 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.938 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.941 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_snapshots 2024-09-18T11:45:23.942 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_snapshots' 2024-09-18T11:45:23.942 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-18T11:45:23.946 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.946 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.946 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.946 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.953 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-18T11:45:23.953 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_pool 2024-09-18T11:45:23.953 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_pool: /' 2024-09-18T11:45:23.953 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_snapshots 2024-09-18T11:45:23.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_snapshots 2024-09-18T11:45:23.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24014 2024-09-18T11:45:23.957 INFO:tasks.workunit.client.0.smithi129.stdout:test api_snapshots on pid 24014 2024-09-18T11:45:23.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_snapshots on pid 24014' 2024-09-18T11:45:23.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24014 2024-09-18T11:45:23.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-18T11:45:23.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_snapshots_pp 2024-09-18T11:45:23.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_snapshots_pp' 2024-09-18T11:45:23.962 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.962 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.963 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.963 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.965 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_snapshots_pp 2024-09-18T11:45:23.965 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_snapshots 2024-09-18T11:45:23.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-18T11:45:23.966 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-18T11:45:23.967 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_snapshots_pp 2024-09-18T11:45:23.967 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24028 2024-09-18T11:45:23.968 INFO:tasks.workunit.client.0.smithi129.stdout:test api_snapshots_pp on pid 24028 2024-09-18T11:45:23.968 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_snapshots_pp on pid 24028' 2024-09-18T11:45:23.968 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24028 2024-09-18T11:45:23.968 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-18T11:45:23.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_stat 2024-09-18T11:45:23.970 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.970 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.970 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_stat' 2024-09-18T11:45:23.970 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.970 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.971 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_snapshots_pp 2024-09-18T11:45:23.971 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-18T11:45:23.971 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-18T11:45:23.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_stat 2024-09-18T11:45:23.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.974 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_stat 2024-09-18T11:45:23.974 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24050 2024-09-18T11:45:23.974 INFO:tasks.workunit.client.0.smithi129.stdout:test api_stat on pid 24050 2024-09-18T11:45:23.975 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_stat on pid 24050' 2024-09-18T11:45:23.975 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24050 2024-09-18T11:45:23.975 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.975 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.975 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-18T11:45:23.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.980 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.980 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_stat 2024-09-18T11:45:23.980 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-18T11:45:23.980 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_stat: /' 2024-09-18T11:45:23.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_stat_pp 2024-09-18T11:45:23.980 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_stat_pp' 2024-09-18T11:45:23.989 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_stat_pp 2024-09-18T11:45:23.989 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.992 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_stat_pp 2024-09-18T11:45:23.992 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24083 2024-09-18T11:45:23.992 INFO:tasks.workunit.client.0.smithi129.stdout:test api_stat_pp on pid 24083 2024-09-18T11:45:23.993 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_stat_pp on pid 24083' 2024-09-18T11:45:23.993 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24083 2024-09-18T11:45:23.993 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:23.993 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:23.993 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-18T11:45:23.993 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_watch_notify 2024-09-18T11:45:23.993 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_watch_notify' 2024-09-18T11:45:23.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_watch_notify 2024-09-18T11:45:23.997 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:23.997 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.997 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:23.997 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:23.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:23.997 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-18T11:45:23.998 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_stat_pp 2024-09-18T11:45:23.998 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-18T11:45:23.999 INFO:tasks.workunit.client.0.smithi129.stdout:test api_watch_notify on pid 24109 2024-09-18T11:45:24.000 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_watch_notify 2024-09-18T11:45:24.000 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24109 2024-09-18T11:45:24.000 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_watch_notify on pid 24109' 2024-09-18T11:45:24.000 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24109 2024-09-18T11:45:24.000 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.000 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.001 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_watch_notify_pp 2024-09-18T11:45:24.001 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_watch_notify_pp' 2024-09-18T11:45:24.001 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-18T11:45:24.001 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.001 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_watch_notify_pp 2024-09-18T11:45:24.007 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_watch_notify_pp 2024-09-18T11:45:24.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24124 2024-09-18T11:45:24.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_watch_notify_pp on pid 24124' 2024-09-18T11:45:24.008 INFO:tasks.workunit.client.0.smithi129.stdout:test api_watch_notify_pp on pid 24124 2024-09-18T11:45:24.009 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24124 2024-09-18T11:45:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-18T11:45:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cmd 2024-09-18T11:45:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cmd' 2024-09-18T11:45:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cmd 2024-09-18T11:45:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_watch_notify 2024-09-18T11:45:24.010 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-18T11:45:24.015 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.015 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.015 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.015 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.015 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-18T11:45:24.017 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-18T11:45:24.017 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-09-18T11:45:24.029 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.029 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-18T11:45:24.031 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cmd 2024-09-18T11:45:24.031 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24176 2024-09-18T11:45:24.031 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cmd on pid 24176' 2024-09-18T11:45:24.032 INFO:tasks.workunit.client.0.smithi129.stdout:test api_cmd on pid 24176 2024-09-18T11:45:24.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24176 2024-09-18T11:45:24.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-18T11:45:24.033 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cmd_pp 2024-09-18T11:45:24.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cmd_pp' 2024-09-18T11:45:24.033 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.033 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cmd_pp 2024-09-18T11:45:24.037 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.037 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.037 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.037 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.038 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-18T11:45:24.038 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cmd_pp 2024-09-18T11:45:24.038 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24199 2024-09-18T11:45:24.039 INFO:tasks.workunit.client.0.smithi129.stdout:test api_cmd_pp on pid 24199 2024-09-18T11:45:24.039 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cmd_pp on pid 24199' 2024-09-18T11:45:24.039 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24199 2024-09-18T11:45:24.039 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.040 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.040 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cmd 2024-09-18T11:45:24.041 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-18T11:45:24.041 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_service 2024-09-18T11:45:24.042 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_service' 2024-09-18T11:45:24.042 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_service 2024-09-18T11:45:24.042 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.044 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_service 2024-09-18T11:45:24.045 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24204 2024-09-18T11:45:24.045 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_service on pid 24204' 2024-09-18T11:45:24.045 INFO:tasks.workunit.client.0.smithi129.stdout:test api_service on pid 24204 2024-09-18T11:45:24.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24204 2024-09-18T11:45:24.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-18T11:45:24.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.047 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_service.log 2024-09-18T11:45:24.047 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.047 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.047 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.047 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.048 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_service_pp 2024-09-18T11:45:24.048 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_service_pp' 2024-09-18T11:45:24.048 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cmd_pp 2024-09-18T11:45:24.048 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.048 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_service_pp 2024-09-18T11:45:24.048 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-18T11:45:24.048 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-18T11:45:24.049 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cmd: /' 2024-09-18T11:45:24.051 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_service: /' 2024-09-18T11:45:24.051 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_service 2024-09-18T11:45:24.052 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_service_pp 2024-09-18T11:45:24.052 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24219 2024-09-18T11:45:24.052 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_service_pp on pid 24219' 2024-09-18T11:45:24.052 INFO:tasks.workunit.client.0.smithi129.stdout:test api_service_pp on pid 24219 2024-09-18T11:45:24.053 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24219 2024-09-18T11:45:24.053 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.053 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.053 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-18T11:45:24.058 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.058 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.059 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.059 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.060 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_service_pp 2024-09-18T11:45:24.061 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_c_write_operations 2024-09-18T11:45:24.061 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_c_write_operations' 2024-09-18T11:45:24.062 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-18T11:45:24.062 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-18T11:45:24.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_c_write_operations 2024-09-18T11:45:24.068 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_c_write_operations 2024-09-18T11:45:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24273 2024-09-18T11:45:24.078 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_c_write_operations on pid 24273' 2024-09-18T11:45:24.078 INFO:tasks.workunit.client.0.smithi129.stdout:test api_c_write_operations on pid 24273 2024-09-18T11:45:24.079 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24273 2024-09-18T11:45:24.079 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.079 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-18T11:45:24.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_c_read_operations 2024-09-18T11:45:24.082 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_c_read_operations' 2024-09-18T11:45:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-18T11:45:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-18T11:45:24.083 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_c_write_operations 2024-09-18T11:45:24.093 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_c_read_operations 2024-09-18T11:45:24.093 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.099 INFO:tasks.workunit.client.0.smithi129.stdout:test api_c_read_operations on pid 24299 2024-09-18T11:45:24.099 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_c_read_operations 2024-09-18T11:45:24.099 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24299 2024-09-18T11:45:24.099 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_c_read_operations on pid 24299' 2024-09-18T11:45:24.100 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24299 2024-09-18T11:45:24.100 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.100 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.100 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-18T11:45:24.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cls_remote_reads 2024-09-18T11:45:24.101 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cls_remote_reads' 2024-09-18T11:45:24.102 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.102 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.102 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.102 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.102 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-18T11:45:24.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cls_remote_reads 2024-09-18T11:45:24.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.103 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_c_read_operations 2024-09-18T11:45:24.103 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-18T11:45:24.104 INFO:tasks.workunit.client.0.smithi129.stdout:test api_cls_remote_reads on pid 24309 2024-09-18T11:45:24.105 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cls_remote_reads 2024-09-18T11:45:24.105 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24309 2024-09-18T11:45:24.105 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cls_remote_reads on pid 24309' 2024-09-18T11:45:24.105 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24309 2024-09-18T11:45:24.105 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.105 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.109 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-18T11:45:24.110 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.110 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.110 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-18T11:45:24.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-18T11:45:24.111 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s list_parallel 2024-09-18T11:45:24.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' list_parallel' 2024-09-18T11:45:24.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo list_parallel 2024-09-18T11:45:24.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.116 INFO:tasks.workunit.client.0.smithi129.stdout:test list_parallel on pid 24327 2024-09-18T11:45:24.116 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=list_parallel 2024-09-18T11:45:24.116 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24327 2024-09-18T11:45:24.116 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test list_parallel on pid 24327' 2024-09-18T11:45:24.116 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24327 2024-09-18T11:45:24.116 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.116 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.117 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-09-18T11:45:24.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s open_pools_parallel 2024-09-18T11:45:24.121 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' open_pools_parallel' 2024-09-18T11:45:24.125 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-18T11:45:24.126 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo open_pools_parallel 2024-09-18T11:45:24.126 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.129 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=open_pools_parallel 2024-09-18T11:45:24.130 INFO:tasks.workunit.client.0.smithi129.stdout:test open_pools_parallel on pid 24377 2024-09-18T11:45:24.130 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24377 2024-09-18T11:45:24.130 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test open_pools_parallel on pid 24377' 2024-09-18T11:45:24.130 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24377 2024-09-18T11:45:24.131 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-18T11:45:24.131 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s delete_pools_parallel 2024-09-18T11:45:24.131 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-18T11:45:24.131 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' delete_pools_parallel' 2024-09-18T11:45:24.131 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-09-18T11:45:24.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-18T11:45:24.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_open_pools_parallel 2024-09-18T11:45:24.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-18T11:45:24.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-18T11:45:24.133 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_list_parallel 2024-09-18T11:45:24.134 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ list_parallel: /' 2024-09-18T11:45:24.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo delete_pools_parallel 2024-09-18T11:45:24.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=delete_pools_parallel 2024-09-18T11:45:24.139 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24402 2024-09-18T11:45:24.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test delete_pools_parallel on pid 24402' 2024-09-18T11:45:24.140 INFO:tasks.workunit.client.0.smithi129.stdout:test delete_pools_parallel on pid 24402 2024-09-18T11:45:24.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=24402 2024-09-18T11:45:24.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ ret=0 2024-09-18T11:45:24.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-09-18T11:45:24.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:45:24.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23760 2024-09-18T11:45:24.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23760 2024-09-18T11:45:24.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-18T11:45:24.141 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -z '' ']' 2024-09-18T11:45:24.142 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.142 INFO:tasks.workunit.client.0.smithi129.stderr:+ case $- in 2024-09-18T11:45:24.142 INFO:tasks.workunit.client.0.smithi129.stderr:+ return 2024-09-18T11:45:24.142 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_delete_pools_parallel 2024-09-18T11:45:24.142 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-18T11:45:24.144 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-18T11:45:24.272 INFO:tasks.workunit.client.0.smithi129.stdout:10 client.4502.objecter _get_latest_version latest 18, have it 2024-09-18T11:45:24.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c61836c0 10 client.4502.objecter create_pool name=ceph_test_rados_api_asio 2024-09-18T11:45:24.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c61836c0 10 client.4502.objecter pool_op_submit 1 2024-09-18T11:45:24.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c61836c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:45:24.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.112:6790/0 -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x5556dc826140 con 0x5556dc894430 2024-09-18T11:45:24.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:23.912+0000 7fd8c0cb6700 10 client.4502.objecter ms_handle_connect 0x7fd8b00362c0 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 10 ==== pool_op_reply(tid 1 (0) Success v19) v1 ==== 43+0+0 (unknown 3885183601 0 0) 0x7fd8b40130c0 con 0x5556dc894430 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 pool_op_reply(tid 1 (0) Success v19) v1 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 10 client.4502.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v19) v1 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 10 client.4502.objecter have request 1 at 0x5556dc92f280 Op: create 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 20 client.4502.objecter waiting for client to reach epoch 19 before calling back 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 10 monclient: _renew_subs 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=19}) v3 -- 0x5556dc8930b0 con 0x5556dc894430 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 10 client.4502.objecter done 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== mon.2 v1:172.21.15.112:6790/0 11 ==== osd_map(19..19 src has 1..19) v4 ==== 3597+0+0 (unknown 1191963694 0 0) 0x7fd8b4013460 con 0x5556dc894430 2024-09-18T11:45:24.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_map(19..19 src has 1..19) v4 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 3 client.4502.objecter handle_osd_map got epochs [19,19] > 18 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 3 client.4502.objecter handle_osd_map decoding incremental epoch 19 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c0cb6700 20 client.4502.objecter dump_active .. 0 homeless 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 10 librados: set snap read head -> 1 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 10 librados: writefull oid=exist nspace= 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc874920 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 19 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:24.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 2 osd.3 2024-09-18T11:45:25.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 2 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 15 client.4502.objecter _send_op 2 to 5.1 on osd.3 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:2 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x5556dc958be0 con 0x5556dc890c00 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.264+0000 7fd8c44bd700 10 monclient: get_auth_request con 0x5556dc890c00 auth_method 0 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:24.268+0000 7fd8c0cb6700 10 client.4502.objecter ms_handle_connect 0x5556dc890c00 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 1 ==== osd_map(20..20 src has 1..20) v4 ==== 8930+0+0 (unknown 57618720 0 0) 0x7fd8bc001b70 con 0x5556dc890c00 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_map(20..20 src has 1..20) v4 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 3 client.4502.objecter handle_osd_map got epochs [20,20] > 19 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 3 client.4502.objecter handle_osd_map decoding incremental epoch 20 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 10 client.4502.objecter checking op 2 2024-09-18T11:45:25.414 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 20 client.4502.objecter dump_active .. 0 homeless 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.396+0000 7fd8c0cb6700 20 client.4502.objecter 2 5.d3a59201 osd.3 exist [writefull 0~5 in=5b] 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 2 ==== osd_op_reply(2 exist [writefull 0~5] v20'1 uv1 ondisk = 0) v8 ==== 149+0+0 (unknown 541397297 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(2 exist [writefull 0~5] v20'1 uv1 ondisk = 0) v8 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 2 ondisk uv 1 in 5.1 attempt 0 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.415 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 2 2024-09-18T11:45:25.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 2 2024-09-18T11:45:25.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 2 2024-09-18T11:45:25.416 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.417 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.404+0000 7fd8c61836c0 10 librados: Objecter returned from writefull r=0 2024-09-18T11:45:25.417 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-18T11:45:25.417 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc874a20 2024-09-18T11:45:25.417 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.417 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.417 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.417 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 3 osd.3 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 3 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _send_op 3 to 5.1 on osd.3 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:3 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc953f90 con 0x5556dc890c00 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc954360 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.418 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 4 osd.3 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 4 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _send_op 4 to 5.1 on osd.3 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:4 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc954850 con 0x5556dc890c00 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 5 client.4502.objecter 2 in flight 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 3638595691 0 3741568377) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.419 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 3 ondisk uv 1 in 5.1 attempt 0 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 5 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 3 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 3 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 3 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 1329292466 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.420 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 4 ondisk uv 0 in 5.1 attempt 0 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 4 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 4 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 4 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc954340 2024-09-18T11:45:25.421 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 5 osd.3 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 5 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _send_op 5 to 5.1 on osd.3 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:5 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc874a20 con 0x5556dc890c00 2024-09-18T11:45:25.422 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc958f90 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 6 osd.3 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 6 2024-09-18T11:45:25.423 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _send_op 6 to 5.1 on osd.3 2024-09-18T11:45:25.424 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:6 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc959480 con 0x5556dc890c00 2024-09-18T11:45:25.424 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.424 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 5 client.4502.objecter 2 in flight 2024-09-18T11:45:25.424 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 3638595691 0 3741568377) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.424 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-18T11:45:25.424 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.424 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 5 ondisk uv 1 in 5.1 attempt 0 2024-09-18T11:45:25.424 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 5 2024-09-18T11:45:25.425 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 5 2024-09-18T11:45:25.425 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 5 2024-09-18T11:45:25.425 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.425 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 5 2024-09-18T11:45:25.425 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.428 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 1329292466 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.428 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-18T11:45:25.428 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.428 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 6 ondisk uv 0 in 5.1 attempt 0 2024-09-18T11:45:25.428 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-18T11:45:25.428 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.428 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 6 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 6 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 6 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (0 ms) 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc958f90 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 7 osd.3 2024-09-18T11:45:25.429 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 7 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _send_op 7 to 5.1 on osd.3 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:7 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc954340 con 0x5556dc890c00 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc979ec0 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 8 osd.3 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.430 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 8 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _send_op 8 to 5.1 on osd.3 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:8 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc97a3b0 con 0x5556dc890c00 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 5 client.4502.objecter 2 in flight 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 3638595691 0 3741568377) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 7 ondisk uv 1 in 5.1 attempt 0 2024-09-18T11:45:25.431 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 5 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 7 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 7 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 7 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 1329292466 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 8 ondisk uv 0 in 5.1 attempt 0 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.432 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 8 2024-09-18T11:45:25.433 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 8 2024-09-18T11:45:25.433 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.433 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-09-18T11:45:25.433 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 8 2024-09-18T11:45:25.433 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-18T11:45:25.433 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 librados: queue_aio_write 0x5556dc81af60 completion 0x5556dc958f90 write_seq 1 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc95a840 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 9 osd.3 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 9 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 15 client.4502.objecter _send_op 9 to 5.1 on osd.3 2024-09-18T11:45:25.434 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:9 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc9590a0 con 0x5556dc890c00 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.408+0000 7fd8c61836c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 9 ==== osd_op_reply(9 exist [write 0~5] v20'2 uv2 ondisk = 0) v8 ==== 149+0+0 (unknown 552823769 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(9 exist [write 0~5] v20'2 uv2 ondisk = 0) v8 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 9 ondisk uv 2 in 5.1 attempt 0 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 9 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 9 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2 ms) 2024-09-18T11:45:25.435 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 9 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c44bd700 20 librados: complete_aio_write 0x5556dc958f90 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 20 librados: queue_aio_write 0x5556dc81af60 completion 0x5556dc95ad30 write_seq 2 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc95a840 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 10 osd.3 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.436 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 10 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 15 client.4502.objecter _send_op 10 to 5.1 on osd.3 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:10 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc9590a0 con 0x5556dc890c00 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.412+0000 7fd8c61836c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 10 ==== osd_op_reply(10 exist [write 0~5] v20'3 uv3 ondisk = 0) v8 ==== 149+0+0 (unknown 1416876045 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(10 exist [write 0~5] v20'3 uv3 ondisk = 0) v8 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 10 ondisk uv 3 in 5.1 attempt 0 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 10 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 10 2024-09-18T11:45:25.437 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.438 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 10 2024-09-18T11:45:25.438 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.438 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c44bd700 20 librados: complete_aio_write 0x5556dc95ad30 2024-09-18T11:45:25.438 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (4 ms) 2024-09-18T11:45:25.438 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-18T11:45:25.438 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-18T11:45:25.438 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 20 librados: queue_aio_write 0x5556dc81af60 completion 0x5556dc95ae40 write_seq 3 2024-09-18T11:45:25.438 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc96d020 2024-09-18T11:45:25.439 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:45:25.439 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.439 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.439 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.439 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 11 osd.3 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 11 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 15 client.4502.objecter _send_op 11 to 5.1 on osd.3 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:11 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc95a8e0 con 0x5556dc890c00 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.416+0000 7fd8c61836c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 11 ==== osd_op_reply(11 exist [write 0~5] v20'4 uv4 ondisk = 0) v8 ==== 149+0+0 (unknown 505185744 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(11 exist [write 0~5] v20'4 uv4 ondisk = 0) v8 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 11 ondisk uv 4 in 5.1 attempt 0 2024-09-18T11:45:25.440 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 11 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 11 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 11 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 20 librados: complete_aio_write 0x5556dc95ae40 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (12 ms) 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc959830 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.441 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 12 osd.3 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 12 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _send_op 12 to 5.1 on osd.3 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:12 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc959d20 con 0x5556dc890c00 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc95a1e0 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.442 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 13 osd.3 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 13 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _send_op 13 to 5.1 on osd.3 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:13 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc96d750 con 0x5556dc890c00 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 5 client.4502.objecter 2 in flight 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 2863821466 0 3741568377) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 12 ondisk uv 4 in 5.1 attempt 0 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 5 2024-09-18T11:45:25.443 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 12 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 12 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 12 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 582179466 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 13 ondisk uv 0 in 5.1 attempt 0 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-18T11:45:25.444 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 13 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 13 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 13 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc95a1e0 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 14 osd.3 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.445 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 14 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _send_op 14 to 5.1 on osd.3 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:14 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc959830 con 0x5556dc890c00 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc96d730 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 15 osd.3 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 15 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _send_op 15 to 5.1 on osd.3 2024-09-18T11:45:25.446 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:15 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc96dc20 con 0x5556dc890c00 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 5 client.4502.objecter 2 in flight 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 2863821466 0 3741568377) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 14 ondisk uv 4 in 5.1 attempt 0 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 5 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 14 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 14 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 14 2024-09-18T11:45:25.447 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 582179466 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 15 ondisk uv 0 in 5.1 attempt 0 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 15 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 15 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 15 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.448 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-09-18T11:45:25.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-18T11:45:25.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc95a1e0 2024-09-18T11:45:25.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 16 osd.3 2024-09-18T11:45:25.449 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 16 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _send_op 16 to 5.1 on osd.3 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:16 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc959830 con 0x5556dc890c00 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc96cdf0 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 17 osd.3 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 17 2024-09-18T11:45:25.450 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _send_op 17 to 5.1 on osd.3 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:17 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc96dc90 con 0x5556dc890c00 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 5 client.4502.objecter 2 in flight 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 2863821466 0 3741568377) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 16 ondisk uv 4 in 5.1 attempt 0 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 5 2024-09-18T11:45:25.451 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 16 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 16 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 5 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 16 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 582179466 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 17 ondisk uv 0 in 5.1 attempt 0 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 17 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 17 2024-09-18T11:45:25.452 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-18T11:45:25.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-18T11:45:25.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 17 2024-09-18T11:45:25.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 librados: queue_aio_write 0x5556dc81af60 completion 0x5556dc959df0 write_seq 4 2024-09-18T11:45:25.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc95a1e0 2024-09-18T11:45:25.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:45:25.453 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.454 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.454 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.454 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 18 osd.3 2024-09-18T11:45:25.454 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 18 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 15 client.4502.objecter _send_op 18 to 5.1 on osd.3 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:18 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc959830 con 0x5556dc890c00 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.428+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 18 ==== osd_op_reply(18 exist [writefull 0~5] v20'5 uv5 ondisk = 0) v8 ==== 149+0+0 (unknown 4155213968 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(18 exist [writefull 0~5] v20'5 uv5 ondisk = 0) v8 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 18 ondisk uv 5 in 5.1 attempt 0 2024-09-18T11:45:25.455 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 18 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 18 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 18 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (6 ms) 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c44bd700 20 librados: complete_aio_write 0x5556dc959df0 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 20 librados: queue_aio_write 0x5556dc81af60 completion 0x5556dc959f00 write_seq 5 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc95a1e0 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:45:25.456 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 19 osd.3 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 19 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 15 client.4502.objecter _send_op 19 to 5.1 on osd.3 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:19 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc959830 con 0x5556dc890c00 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.457 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.436+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 19 ==== osd_op_reply(19 exist [writefull 0~5] v20'6 uv6 ondisk = 0) v8 ==== 149+0+0 (unknown 1781155052 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(19 exist [writefull 0~5] v20'6 uv6 ondisk = 0) v8 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 19 ondisk uv 6 in 5.1 attempt 0 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 19 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 19 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (6 ms) 2024-09-18T11:45:25.458 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 19 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c44bd700 20 librados: complete_aio_write 0x5556dc959f00 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 20 librados: queue_aio_write 0x5556dc81af60 completion 0x5556dc959df0 write_seq 6 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 10 client.4502.objecter _op_submit op 0x5556dc95a1e0 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 20 client.4502.objecter _calc_target epoch 20 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 20 client.4502.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 10 client.4502.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [3,0] primary 3 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 20 client.4502.objecter _get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.459 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 10 client.4502.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 20 osd.3 2024-09-18T11:45:25.460 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 20 client.4502.objecter get_session s=0x5556dc7f2e10 osd=3 3 2024-09-18T11:45:25.460 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 15 client.4502.objecter _session_op_assign 3 20 2024-09-18T11:45:25.460 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 15 client.4502.objecter _send_op 20 to 5.1 on osd.3 2024-09-18T11:45:25.460 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 --> v1:172.21.15.129:6804/21049 -- osd_op(unknown.0.0:20 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x5556dc959830 con 0x5556dc890c00 2024-09-18T11:45:25.460 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.460 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.440+0000 7fd8c61836c0 5 client.4502.objecter 1 in flight 2024-09-18T11:45:25.460 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 <== osd.3 v1:172.21.15.129:6804/21049 20 ==== osd_op_reply(20 exist [writefull 0~5] v20'7 uv7 ondisk = 0) v8 ==== 149+0+0 (unknown 514522936 0 0) 0x7fd8bc001e30 con 0x5556dc890c00 2024-09-18T11:45:25.460 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 10 client.4502.objecter ms_dispatch 0x5556dc7f8790 osd_op_reply(20 exist [writefull 0~5] v20'7 uv7 ondisk = 0) v8 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 10 client.4502.objecter in handle_osd_op_reply 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 7 client.4502.objecter handle_osd_op_reply 20 ondisk uv 7 in 5.1 attempt 0 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 10 client.4502.objecter op 0 rval 0 len 0 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 15 client.4502.objecter handle_osd_op_reply completed tid 20 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 15 client.4502.objecter _finish_op 20 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 4 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 15 client.4502.objecter _session_op_remove 3 20 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 5 client.4502.objecter 0 in flight 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 20 librados: complete_aio_write 0x5556dc959df0 2024-09-18T11:45:25.461 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 10 librados: watch_flush enter 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 10 librados: watch_flush exit 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 10 client.4502.objecter close_session for osd.3 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 >> v1:172.21.15.129:6804/21049 conn(0x5556dc890c00 legacy=0x5556dc9574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 20 client.4502.objecter put_session s=0x5556dc7f2e10 osd=3 1 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 20 client.4502.objecter shutdown clearing up homeless session... 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 10 client.4502.objecter successfully canceled tick 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 >> v1:172.21.15.129:6812/20806 conn(0x7fd8b00362c0 legacy=0x7fd8b0038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 10 monclient: shutdown 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 >> v1:172.21.15.112:6790/0 conn(0x5556dc894430 legacy=0x5556dc86d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c44bd700 1 -- 172.21.15.129:0/1652477188 reap_dead start 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 shutdown_connections 2024-09-18T11:45:25.462 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 >> 172.21.15.129:0/1652477188 conn(0x5556dc899690 msgr2=0x5556dc87bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:45:25.463 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 shutdown_connections 2024-09-18T11:45:25.463 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 1 -- 172.21.15.129:0/1652477188 wait complete. 2024-09-18T11:45:25.463 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.444+0000 7fd8c61836c0 1 librados: shutdown 2024-09-18T11:45:25.463 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] 12 tests from AsioRados (37 ms total) 2024-09-18T11:45:25.463 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-09-18T11:45:25.463 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] Global test environment tear-down 2024-09-18T11:45:25.463 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1555 ms total) 2024-09-18T11:45:25.463 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-18T11:45:26.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-18T11:45:26.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] Global test environment set-up. 2024-09-18T11:45:26.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-18T11:45:26.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-18T11:45:26.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjects (164 ms) 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (54 ms) 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo2,foo3 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo2 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo3 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo4,foo5 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo4 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo5 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo6,foo7 2024-09-18T11:45:26.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo7 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo6 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo4 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo5 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo7 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo6 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo1 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo1 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo2 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo3 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (155 ms) 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-18T11:45:26.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 1 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 10 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 13 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 7 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 14 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 0 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 15 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 11 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 5 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 8 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 6 0 2024-09-18T11:45:26.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 3 0 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 4 0 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 12 0 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 9 0 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2 0 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (167 ms) 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: x cursor=MIN 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2024-09-18T11:45:26.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2024-09-18T11:45:26.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:566253c9:::13:head 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-09-18T11:45:26.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=14 cursor=7:62a1935d:::14:head 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:863748b0:::15:head 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2024-09-18T11:45:26.166 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2024-09-18T11:45:26.167 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:e960b815:::9:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-09-18T11:45:26.168 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=10 expected=10 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=3 expected=3 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=6 expected=6 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-09-18T11:45:26.169 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=8 expected=8 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=5 expected=5 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=11 expected=11 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:863748b0:::15:head 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2024-09-18T11:45:26.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-09-18T11:45:26.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=15 expected=15 2024-09-18T11:45:26.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-09-18T11:45:26.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2024-09-18T11:45:26.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-09-18T11:45:26.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=2 expected=2 2024-09-18T11:45:26.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:e960b815:::9:head 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: ap api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (3146 ms) 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (3146 ms total) 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (3146 ms total) 2024-09-18T11:45:27.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-18T11:45:27.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (45 ms) 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (51 ms) 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (3156 ms) 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3252 ms total) 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-18T11:45:27.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3252 ms total) 2024-09-18T11:45:27.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-18T11:45:27.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-18T11:45:27.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-09-18T11:45:27.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-18T11:45:27.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-09-18T11:45:27.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (126 ms) 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (67 ms) 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (30 ms) 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (24 ms) 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (19 ms) 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (12 ms) 2024-09-18T11:45:27.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (8 ms) 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (8 ms) 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (9 ms) 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (8 ms) 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (42 ms) 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (28 ms) 2024-09-18T11:45:27.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-18T11:45:27.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (21 ms) 2024-09-18T11:45:27.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-18T11:45:27.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (6 ms) 2024-09-18T11:45:27.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (414 ms total) 2024-09-18T11:45:27.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: 2024-09-18T11:45:27.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-18T11:45:27.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (3194 ms total) 2024-09-18T11:45:27.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-09-18T11:45:27.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-18T11:45:27.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-18T11:45:27.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-18T11:45:27.411 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (64 ms) 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (64 ms) 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (3131 ms) 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.327314+0000 mon.b [INF] from='client.4604 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24283-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.327562+0000 mon.b [INF] from='client.4553 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-24023-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.327693+0000 mon.b [INF] from='client.4550 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-24037-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.327850+0000 mon.b [INF] from='client.4565 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-24097-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.328010+0000 mon.b [INF] from='client.4571 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-24125-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.412 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.328189+0000 mon.b [INF] from='client.4625 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-24306-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.328386+0000 mon.b [INF] from='client.4643 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24209-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.328772+0000 mon.b [INF] from='client.4673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24318-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.328979+0000 mon.b [INF] from='client.4679 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24195-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.329247+0000 mon.b [INF] from='client.4487 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-23778-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.329433+0000 mon.b [INF] from='client.4493 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi129-23773-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.329687+0000 mon.b [INF] from='client.4562 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-24060-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:27.413 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.338561+0000 mon.b [WRN] Health check failed: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T11:45:28.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: go list_parallel: process_1_[24424]: starting. 2024-09-18T11:45:28.032 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[24424]: creating pool ceph_test_rados_list_parallel.smithi129-24373 2024-09-18T11:45:28.032 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[24424]: created object 0... 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[24424]: created object 25... 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[24424]: created object 49... 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[24424]: finishing. 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[24424]: shutting down. 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[24426]: starting. 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[24426]: listing objects. 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[24426]: listed object 0... 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[24426]: listed object 25... 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[24426]: saw 50 objects 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[24426]: shutting down. 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[24722]: starting. 2024-09-18T11:45:28.033 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[24722]: creating pool ceph_test_rados_list_parallel.smithi129-24373 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[24722]: created object 0... 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[24722]: created object 25... 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[24722]: created object 49... 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[24722]: finishing. 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[24722]: shutting down. 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[24723]: starting. 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[24723]: listing objects. 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[24723]: listed object 0... 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[24723]: listed object 25... 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[24723]: saw 44 objects 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[24723]: shutting down. 2024-09-18T11:45:28.034 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[24724]: starting. 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[24724]: removed 25 objects... 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[24724]: removed half of the objects 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[24724]: removed 50 objects... 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[24724]: removed 50 objects 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[24724]: shutting down. 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[24773]: starting. 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[24773]: creating pool ceph_test_rados_list_parallel.smithi129-24373 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[24773]: created object 0... 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[24773]: created object 25... 2024-09-18T11:45:28.035 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[24773]: created object 49... 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[24773]: finishing. 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[24773]: shutting down. 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[24774]: starting. 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[24774]: listing objects. 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[24774]: listed object 0... 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[24774]: listed object 25... 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[24774]: listed object 50... 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[24774]: saw 56 objects 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[24774]: shutting down. 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.036 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[24775]: starting. 2024-09-18T11:45:28.037 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[24775]: added 25 objects... 2024-09-18T11:45:28.037 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[24775]: added half of the objects 2024-09-18T11:45:28.037 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[24775]: added 50 objects... 2024-09-18T11:45:28.037 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[24775]: added 50 objects 2024-09-18T11:45:28.037 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[24775]: shutting down. 2024-09-18T11:45:28.037 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.037 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:28.705 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: **********************t: 2024-09-18T11:45:26.357444+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/2393770789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-23975-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:28.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.449415+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/307719865' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-23789-22"}]: dispatch 2024-09-18T11:45:28.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.535985+0000 mon.b [INF] from='client.4703 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-23789-22"}]: dispatch 2024-09-18T11:45:28.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.538234+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/307719865' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-23789-22"}]: dispatch 2024-09-18T11:45:28.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.593792+0000 mon.b [INF] from='client.4703 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-23789-22"}]: dispatch 2024-09-18T11:45:28.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.602475+0000 mon.b [INF] from='client.4703 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-23789-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-18T11:45:28.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:26.603043+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/307719865' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-23789-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.293697+0000 mon.b [INF] from='client.4583 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-24150-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-24150-1"}]': finished 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.293809+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/2393770789' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-23975-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.294507+0000 mon.b [INF] from='client.4703 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-23789-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.305071+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/307719865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-23789-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-23789-22"}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.306700+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/1627309414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24000-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.389236+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/761984175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-23765-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.389833+0000 mon.b [INF] from='client.4703 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-23789-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-23789-22"}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.390537+0000 mon.b [INF] from='client.4697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24000-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.407289+0000 client.admin [INF] onexx 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.408320+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/4232222489' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.421299+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3629606501' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-24060-6"}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.425029+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3629606501' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-24060-6"}]: dispatch 2024-09-18T11:45:28.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.426073+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3629606501' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-24060-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.431674+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3679480750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-24097-7"}]: dispatch 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.434876+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/1236044394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-23778-16"}]: dispatch 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.484451+0000 mon.b [INF] from='client.4796 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-23778-16"}]: dispatch 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.484649+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3679480750' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-24097-7"}]: dispatch 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:27.548940+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3679480750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-24097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.301965+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/761984175' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-23765-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.349213+0000 mon.b [INF] from='client.4697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24000-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.349434+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3629606501' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-24060-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.349601+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3679480750' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-24097-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-18T11:45:28.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.350864+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3629606501' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-24060-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-24060-6"}]: dispatch 2024-09-18T11:45:29.552 INFO:tasks.workunit.client.0.smithi129.stdout: ********* 2024-09-18T11:45:29.552 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[25015]: starting. 2024-09-18T11:45:29.552 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[25015]: creating pool ceph_test_rados_list_parallel.smithi129-24373 2024-09-18T11:45:29.552 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[25015]: created object 0... 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[25015]: created object 25... 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[25015]: created object 49... 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[25015]: finishing. 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[25015]: shutting down. 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[25016]: starting. 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[25016]: listing objects. 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[25016]: listed object 0... 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[25016]: listed object 25... 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[25016]: listed object 50... 2024-09-18T11:45:29.553 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[25016]: listed object 75... 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[25016]: saw 96 objects 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[25016]: shutting down. 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[25019]: starting. 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[25019]: removed 25 objects... 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[25019]: removed half of the objects 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[25019]: removed 50 objects... 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[25019]: removed 50 objects 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[25019]: shutting down. 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.554 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[25018]: starting. 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[25018]: added 25 objects... 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[25018]: added half of the objects 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[25018]: added 50 objects... 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[25018]: added 50 objects 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[25018]: shutting down. 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[25017]: starting. 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[25017]: added 25 objects... 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[25017]: added half of the objects 2024-09-18T11:45:29.555 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[25017]: added 50 objects... 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[25017]: added 50 objects 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[25017]: shutting down. 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[25100]: starting. 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[25100]: creating pool ceph_test_rados_list_parallel.smithi129-24373 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[25100]: created object 0... 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[25100]: created object 25... 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[25100]: created object 49... 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[25100]: finishing. 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[25100]: shutting down. 2024-09-18T11:45:29.556 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.557 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.557 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.557 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:29.557 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: starting. 2024-09-18T11:45:29.557 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: listing objects. 2024-09-18T11:45:29.557 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: listed object 0... 2024-09-18T11:45:30.448 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: listed object 25... 2024-09-18T11:45:30.448 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: listed object 50... 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: listed object 75... 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: listed object 100... 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: listed object 125... 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: saw 150 objects 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[25101]: shutting down. 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[25102]: starting. 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[25102]: added 25 objects... 2024-09-18T11:45:30.449 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[25102]: added half of the objects 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[25102]: added 50 objects... 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[25102]: added 50 objects 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[25102]: shutting down. 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-09-18T11:45:30.450 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******* SUCCESS ********** 2024-09-18T11:45:30.460 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[24414]: starting. 2024-09-18T11:45:30.460 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[24414]: creating pool ceph_test_rados_open_pools_parallel.smithi129-24389 2024-09-18T11:45:30.460 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[24414]: created object 0... 2024-09-18T11:45:30.460 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[24414]: created object 25... 2024-09-18T11:45:30.460 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[24414]: created object 49... 2024-09-18T11:45:30.460 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[24414]: finishing. 2024-09-18T11:45:30.460 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[24414]: shutting down. 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[24415]: starting. 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[24415]: rados_pool_create. 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[24415]: rados_ioctx_create. 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[24415]: shutting down. 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[24825]: starting. 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[24825]: creating pool ceph_test_rados_open_pools_parallel.smithi129-24389 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[24825]: created object 0... 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[24825]: created object 25... 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[24825]: created object 49... 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[24825]: finishing. 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[24825]: shutting down. 2024-09-18T11:45:30.461 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[24826]: starting. 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[24826]: rados_pool_create. 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[24826]: rados_ioctx_create. 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[24826]: shutting down. 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[24427]: starting. 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[24427]: creating pool ceph_test_rados_delete_pools_parallel.smithi129-24407 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[24427]: created object 0... 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[24427]: created object 25... 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[24427]: created object 49... 2024-09-18T11:45:30.462 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[24427]: finishing. 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[24427]: shutting down. 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[24428]: starting. 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[24428]: deleting pool ceph_test_rados_delete_pools_parallel.smithi129-24407 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[24428]: shutting down. 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[24827]: starting. 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[24827]: creating pool ceph_test_rados_delete_pools_parallel.smithi129-24407 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[24827]: created object 0... 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[24827]: created object 25... 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[24827]: created object 49... 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[24827]: finishing. 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[24827]: shutting down. 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-09-18T11:45:30.463 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[24829]: starting. 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[24829]: listing objects. 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[24829]: listed object 0... 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[24829]: listed object 25... 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[24829]: saw 50 objects 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[24829]: shutting down. 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[24828]: starting. 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[24828]: deleting pool ceph_test_rados_delete_pools_parallel.smithi129-24407 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[24828]: shutting down. 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-09-18T11:45:30.464 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: seed 23789 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (111 ms) 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-18T11:45:30.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (16 ms) 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (10 ms) 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (10 ms) 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (4 ms) 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (6 ms) 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (8 ms) 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-18T11:45:30.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (7 ms) 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (7 ms) 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2 ms) 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (7 ms) 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (9 ms) 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-09-18T11:45:30.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (19 ms) 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (10 ms) 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (13 ms) 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (7 ms) 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (261 ms total) 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-18T11:45:30.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1094 ms) 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (17 ms) 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (6 ms) 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (11 ms) 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-18T11:45:30.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (15 ms) 2024-09-18T11:45:31.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN api_cmd: got: 2024-09-18T11:45:28.351215+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/1236044394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-23778-16"}]: dispatch 2024-09-18T11:45:31.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.351429+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3679480750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-24097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-24097-7"}]: dispatch 2024-09-18T11:45:31.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.354287+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/2393770789' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-23975-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.361292+0000 mon.b [INF] from='client.4796 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-23778-16"}]: dispatch 2024-09-18T11:45:31.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.365246+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/1236044394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-23778-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-18T11:45:31.710 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.366076+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/1646755849' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24283-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.366662+0000 mon.b [INF] from='client.4796 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-23778-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.366860+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/245650987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-23773-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.367953+0000 mon.b [INF] from='client.4784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24283-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:28.368112+0000 mon.b [INF] from='client.4790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-23773-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:29.707594+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/4232222489' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.041354+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.042924+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.044398+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.045953+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.047294+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.048634+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.049858+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-09-18T11:45:31.711 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.051189+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.052475+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.053554+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/2552865429' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.317620+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3629606501' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-24060-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-24060-6"}]': finished 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.317747+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3679480750' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-24097-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-24097-7"}]': finished 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.406838+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/2393770789' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-23975-4", "tierpool": "test-rados-api-smithi129-23975-4-cache"}]': finished 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.439111+0000 mon.b [INF] from='client.4583 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-24150-1"}]': finished 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.441422+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/2393770789' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-23975-4", "overlaypool": "test-rados-api-smithi129-23975-4-cache"}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.441788+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/3886812262' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-24150-1"}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.442972+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/3151349753' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-23765-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.443572+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/878942026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24000-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.445493+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/245650987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-23773-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.446021+0000 mon.b [INF] from='client.4583 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-24150-1"}]: dispatch 2024-09-18T11:45:31.712 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.446524+0000 mon.b [INF] from='client.4964 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-23765-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.448387+0000 mon.b [INF] from='client.4967 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24000-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:30.448713+0000 mon.b [INF] from='client.4790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-23773-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.324192+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/1910578365' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-23859-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-23859-10"}]': finished 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.324380+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/3494192635' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-23818-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-23818-10"}]': finished 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.324505+0000 mon.b [INF] from='client.4796 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-23778-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-23778-16"}]': finished 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.324676+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/2393770789' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-23975-4", "overlaypool": "test-rados-api-smithi129-23975-4-cache"}]': finished 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.324793+0000 mon.b [INF] from='client.4583 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-24150-1"}]': finished 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.324988+0000 mon.b [INF] from='client.4964 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-23765-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.325176+0000 mon.b [INF] from='client.4967 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24000-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.325290+0000 mon.b [INF] from='client.4790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-23773-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-18T11:45:31.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.330047+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/2393770789' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-23975-4-cache", "mode": "writeback"}]: dispatch 2024-09-18T11:45:31.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.330803+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/245650987' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-23773-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-18T11:45:31.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.331555+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/307719865' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-23789-22"}]: dispatch 2024-09-18T11:45:31.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.333013+0000 mon.c [INF] from='client.? v1:172.21.15.129:0/156445787' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24283-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:33.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.333611+0000 mon.b [INF] from='client.4790 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smit ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-18T11:45:33.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (11 ms) 2024-09-18T11:45:33.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (11 ms) 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (8 ms) 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (7 ms) 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (11 ms) 2024-09-18T11:45:33.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (21 ms) 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (12 ms) 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8 ms) 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (7 ms) 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1254 ms total) 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: 2024-09-18T11:45:33.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-18T11:45:33.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (9479 ms total) 2024-09-18T11:45:33.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-18T11:45:33.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] Global test environment set-up. 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5061 ms) 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.RegisterLate (36 ms) 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 5 pool(s) do not have an application enabled 2024-09-18T11:45:33.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum b,c (age 44s), out of quorum: a 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: x(active, since 75s) 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 6 osds: 6 up (since 70s), 6 in (since 77s) 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 16 pools, 433 pgs 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 1.7 GiB used, 538 GiB / 540 GiB avail 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 70.208% pgs unknown 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 27.714% pgs not active 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 304 unknown 2024-09-18T11:45:33.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 120 creating+peering 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 9 active+clean 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 6 pool(s) do not have an application enabled 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum b,c (age 46s), out of quorum: a 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: x(active, since 77s) 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 6 osds: 6 up (since 72s), 6 in (since 79s) 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: foo: 14 portals active (1 hosts, 3 zones) 2024-09-18T11:45:33.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 18 pools, 497 pgs 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 38 objects, 581 KiB 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 1.7 GiB used, 538 GiB / 540 GiB avail 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 62.374% pgs unknown 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 16.901% pgs not active 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 310 unknown 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 103 active+clean 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 84 creating+peering 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.604 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: io: 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: client: 0 B/s rd, 21 KiB/s wr, 0 op/s rd, 30 op/s wr 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 1 cache pools are missing hit_sets 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 9 pool(s) do not have an application enabled 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum b,c (age 48s), out of quorum: a 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: x(active, since 79s) 2024-09-18T11:45:33.605 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 6 osds: 6 up (since 74s), 6 in (since 81s) 2024-09-18T11:45:33.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-18T11:45:33.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-18T11:45:33.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:33.606 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: seed 24097 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (180 ms) 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (13 ms) 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-09-18T11:45:34.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (10 ms) 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (48 ms) 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (251 ms total) 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (967 ms) 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (25 ms) 2024-09-18T11:45:34.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (7 ms) 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (17 ms) 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1016 ms total) 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10344 ms total) 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: Running main() from gmock_main.cc 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] Global test environment set-up. 2024-09-18T11:45:34.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.Stat (187 ms) 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.StatNS (32 ms) 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 4 tests from LibRadosStat (223 ms total) 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-18T11:45:34.358 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (949 ms) 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (28 ms) 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (13 ms) 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (9 ms) 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1000 ms total) 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] Global test environment tear-down 2024-09-18T11:45:34.359 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (10355 ms total) 2024-09-18T11:45:34.360 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ PASSED ] 8 tests. 2024-09-18T11:45:34.739 INFO:tasks.workunit.client.0.smithi129.stdout:hi129-23773-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-18T11:45:34.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.346521+0000 mon.b [INF] from='client.4703 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-23789-22"}]: dispatch 2024-09-18T11:45:34.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.346810+0000 mon.b [INF] from='client.4985 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-24283-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-18T11:45:34.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.708381+0000 client.admin [INF] threexx 2024-09-18T11:45:34.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-09-18T11:45:31.708815+0000 mon.b [INF] from='client.? v1:172.21.15.129:0/4232222489' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-18T11:45:34.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7430 ms) 2024-09-18T11:45:34.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10689 ms total) 2024-09-18T11:45:34.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: 2024-09-18T11:45:34.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-18T11:45:34.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10689 ms total) 2024-09-18T11:45:34.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-18T11:45:35.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: Running main() from gmock_main.cc 2024-09-18T11:45:35.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-18T11:45:35.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] Global test environment set-up. 2024-09-18T11:45:35.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-18T11:45:35.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-18T11:45:35.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (137 ms) 2024-09-18T11:45:35.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (71 ms) 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (21 ms) 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.Checksum (3 ms) 2024-09-18T11:45:35.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (8 ms) 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (11 ms) 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (13 ms) 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (3 ms) 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.TruncTest (11 ms) 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-18T11:45:35.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (7 ms) 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RmXattr (28 ms) 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.XattrIter (13 ms) 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 14 tests from LibRadosIo (330 ms total) 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1078 ms) 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (19 ms) 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (14 ms) 2024-09-18T11:45:35.355 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (14 ms) 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (15 ms) 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (18 ms) 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (9 ms) 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (9 ms) 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (30 ms) 2024-09-18T11:45:35.356 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-18T11:45:35.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (12 ms) 2024-09-18T11:45:35.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1218 ms total) 2024-09-18T11:45:35.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: 2024-09-18T11:45:35.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] Global test environment tear-down 2024-09-18T11:45:35.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [==========] 24 tests from 2 test suites ran. (11491 ms total) 2024-09-18T11:45:35.357 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ PASSED ] 24 tests. 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: seed 23859 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (101 ms) 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (9 ms) 2024-09-18T11:45:37.377 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1057 ms) 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1062 ms) 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (10 ms) 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (7 ms) 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (16 ms) 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (12 ms) 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2274 ms total) 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-18T11:45:37.378 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1074 ms) 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (14 ms) 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1019 ms) 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1055 ms) 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (17 ms) 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (20 ms) 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (19 ms) 2024-09-18T11:45:37.379 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (17 ms) 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3235 ms total) 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13468 ms total) 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: Running main() from gmock_main.cc 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] Global test environment set-up. 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (111 ms) 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-18T11:45:37.380 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockShared (13 ms) 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1065 ms) 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1046 ms) 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (9 ms) 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.Unlock (11 ms) 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (18 ms) 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (5 ms) 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLock (2278 ms total) 2024-09-18T11:45:37.381 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1064 ms) 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (19 ms) 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1013 ms) 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1055 ms) 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (36 ms) 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (13 ms) 2024-09-18T11:45:37.382 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-18T11:45:37.383 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (22 ms) 2024-09-18T11:45:37.383 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-18T11:45:37.383 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (13 ms) 2024-09-18T11:45:37.383 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3235 ms total) 2024-09-18T11:45:37.383 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: 2024-09-18T11:45:37.383 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] Global test environment tear-down 2024-09-18T11:45:37.383 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13483 ms total) 2024-09-18T11:45:37.383 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-18T11:45:40.599 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1090 ms) 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (16 ms) 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1106 ms total) 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: 2024-09-18T11:45:40.600 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (61 ms) 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4028 ms) 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (17 ms) 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (7 ms) 2024-09-18T11:45:40.601 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070574672 notify_id 137438953477 notifier_gid 4988 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (17 ms) 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070577824 notify_id 137438953472 notifier_gid 4988 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (23 ms) 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070555472 notify_id 137438953477 notifier_gid 4988 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (18 ms) 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070555472 notify_id 137438953473 notifier_gid 4988 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (24 ms) 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-18T11:45:40.602 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070555472 notify_id 137438953478 notifier_gid 4988 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (19 ms) 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070558256 notify_id 137438953472 notifier_gid 4988 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (32 ms) 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: trying... 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070558256 notify_id 137438953474 notifier_gid 4988 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: timed out 2024-09-18T11:45:40.603 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2024-09-18T11:45:45.388 INFO:tasks.workunit.client.0.smithi129.stdout: api_watc api_watch_notify: Running main() from gmock_main.cc 2024-09-18T11:45:45.388 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-18T11:45:45.388 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-18T11:45:45.388 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-18T11:45:45.388 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify_test_cb 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (198 ms) 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94211083719520 err -107 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1015 ms) 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94211083719520 err -107 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1009 ms) 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-18T11:45:45.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4571 notify_id 98784247808 cookie 94211083719520 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (16 ms) 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4571 notify_id 98784247809 cookie 94211083732624 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (16 ms) 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4571 notify_id 98784247809 cookie 94211083732624 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (15 ms) 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4571 notify_id 98784247809 cookie 94211083751392 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4571 notify_id 98784247809 cookie 94211083779968 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (18 ms) 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4571 notify_id 98784247810 cookie 94211083751392 2024-09-18T11:45:45.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4571 notify_id 103079215107 cookie 94211083751392 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3012 ms) 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94211083753648 err -107 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 4571 notify_id 133143986180 cookie 94211083753648 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5021 ms) 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94211083751392 err -107 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1017 ms) 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11339 ms total) 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-18T11:45:45.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify_test_cb 2024-09-18T11:45:45.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1137 ms) 2024-09-18T11:45:45.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1137 ms total) 2024-09-18T11:45:45.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: 2024-09-18T11:45:45.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-18T11:45:45.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (21365 ms total) 2024-09-18T11:45:45.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-18T11:45:45.400 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-18T11:45:45.400 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-18T11:45:45.400 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-18T11:45:45.400 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-18T11:45:45.400 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (3220 ms) 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3136 ms) 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2898 ms) 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3016 ms) 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-18T11:45:45.401 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2974 ms) 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3019 ms) 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3043 ms) 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (21306 ms total) 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (21306 ms total) 2024-09-18T11:45:45.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-18T11:45:53.277 INFO:tasks.workunit.client.0.smithi129.stdout:h_notify_pp: flushed 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3013 ms) 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: trying... 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070558256 notify_id 150323855363 notifier_gid 4988 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: timed out 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushed 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3016 ms) 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: List watches 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 2024-09-18T11:45:53.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070558256 notify_id 163208757249 notifier_gid 4988 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 done 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: watch_check 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: unwatch2 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: done 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3026 ms) 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: List watches 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94879070585056 notify_id 176093659138 notifier_gid 4988 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 done 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: watch_check 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: unwatch2 2024-09-18T11:45:53.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2024-09-18T11:45:53.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: done 2024-09-18T11:45:53.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3027 ms) 2024-09-18T11:45:53.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16328 ms total) 2024-09-18T11:45:53.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: 2024-09-18T11:45:53.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-18T11:45:53.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (29234 ms total) 2024-09-18T11:45:53.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-18T11:45:54.679 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 23 pools, 561 pgs 2024-09-18T11:45:54.679 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 57 objects, 121 MiB 2024-09-18T11:45:54.679 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 1.7 GiB used, 538 GiB / 540 GiB avail 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 42.424% pgs unknown 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 3.743% pgs not active 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 302 active+clean 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 238 unknown 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 21 creating+peering 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: io: 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: client: 12 MiB/s rd, 31 MiB/s wr, 254 op/s rd, 290 op/s wr 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4462 ms) 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.Status (21039 ms) 2024-09-18T11:45:54.680 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] 4 tests from LibRadosService (30598 ms total) 2024-09-18T11:45:54.681 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-09-18T11:45:54.681 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] Global test environment tear-down 2024-09-18T11:45:54.681 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [==========] 4 tests from 1 test suite ran. (30598 ms total) 2024-09-18T11:45:54.681 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ PASSED ] 4 tests. 2024-09-18T11:45:55.953 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-18T11:45:55.953 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-18T11:45:55.953 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-18T11:45:55.953 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-18T11:45:55.953 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5038 ms) 2024-09-18T11:45:55.953 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-18T11:45:55.953 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (65 ms) 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (21064 ms) 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: attempt 0 of 20 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5696 ms) 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31863 ms total) 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31863 ms total) 2024-09-18T11:45:55.954 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-18T11:46:01.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: Running main() from gmock_main.cc 2024-09-18T11:46:01.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-18T11:46:01.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] Global test environment set-up. 2024-09-18T11:46:01.361 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2373 ms) 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3081 ms) 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2920 ms) 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3017 ms) 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2899 ms) 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5135 ms) 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-18T11:46:01.362 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5412 ms) 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4473 ms) 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8082 ms) 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] 9 tests from LibRadosPools (37392 ms total) 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] Global test environment tear-down 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37392 ms total) 2024-09-18T11:46:01.363 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-18T11:46:03.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-18T11:46:03.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] Global test environment set-up. 2024-09-18T11:46:03.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-18T11:46:03.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-18T11:46:03.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-18T11:46:03.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-18T11:46:03.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2024-09-18T11:45:23.952+0000 7f7a6bb656c0 -1 failed for service _ceph-mon._tcp 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: ceph-mon 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-09-18T11:45:23.952+0000 7f7a6bb656c0 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (46 ms) 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5012 ms) 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5059 ms total) 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-18T11:46:03.307 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x559368c644b0 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x559368c68cb0 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 2 aios 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x559368c644b0 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x559368c68cb0 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: done. 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5386 ms) 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5386 ms total) 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-18T11:46:03.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.Exec (41 ms) 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (20 ms) 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (14 ms) 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4896 ms) 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-18T11:46:03.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21362 ms) 2024-09-18T11:46:03.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 8 tests from LibRadosMisc (26333 ms total) 2024-09-18T11:46:03.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-09-18T11:46:03.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] Global test environment tear-down 2024-09-18T11:46:03.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (39379 ms total) 2024-09-18T11:46:03.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2032 ms) 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2078 ms) 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-18T11:46:15.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1893 ms) 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2011 ms) 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8015 ms total) 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4099 ms) 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3968 ms) 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (8067 ms total) 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-18T11:46:15.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3071 ms) 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1270 ms) 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2182 ms) 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1815 ms) 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8338 ms total) 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4023 ms) 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-18T11:46:15.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4003 ms) 2024-09-18T11:46:15.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8026 ms total) 2024-09-18T11:46:15.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2024-09-18T11:46:15.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-18T11:46:15.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (51290 ms total) 2024-09-18T11:46:15.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-09-18T11:46:15.878 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T11:46:15.878 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T11:46:15.879 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T11:46:15.879 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T11:46:15.882 INFO:tasks.ceph.mon.a:Started 2024-09-18T11:46:15.882 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:46:15.882 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:46:15.974 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:15.971+0000 7f934d610a00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 22 -> 0 (sync_force? mon rebuild?) 2024-09-18T11:46:15.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient: get_monmap_and_config 2024-09-18T11:46:15.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient: build_initial_monmap 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient: monmap: 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:15.985641+0000 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:15.985641+0000 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:15.986 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:15.987 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 1 Processor -- start 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 1 -- start start 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient: init 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient(hunting): picked mon.noname-a con 0x7f61f00a2270 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient(hunting): picked mon.noname-b con 0x7f61f006aa70 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient(hunting): picked mon.noname-c con 0x7f61f006cb80 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:15.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f006a830 con 0x7f61f00a2270 2024-09-18T11:46:15.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f00a2640 con 0x7f61f006cb80 2024-09-18T11:46:15.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f006d000 con 0x7f61f006aa70 2024-09-18T11:46:15.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:15.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f74ee700 10 monclient(hunting): authenticate will time out at 947.827462s 2024-09-18T11:46:15.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f528a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f61f00a2270 0x7f61f00a0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38810/0 (socket says 172.21.15.112:38810) 2024-09-18T11:46:15.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.983+0000 7f61f528a700 1 -- 172.21.15.112:0/2256518547 learned_addr learned my addr 172.21.15.112:0/2256518547 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:15.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f528a700 1 -- 172.21.15.112:0/2256518547 >> v1:172.21.15.112:6789/0 conn(0x7f61f00a2270 legacy=0x7f61f00a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:15.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f5a8b700 1 -- 172.21.15.112:0/2256518547 >> v1:172.21.15.129:6789/0 conn(0x7f61f006cb80 legacy=0x7f61f0137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:15.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872568469 0 0) 0x7f61f00a2640 con 0x7f61f006cb80 2024-09-18T11:46:15.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:15.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:15.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient(hunting): my global_id is 17359 2024-09-18T11:46:15.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61dc003570 con 0x7f61f006cb80 2024-09-18T11:46:15.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3378316717 0 0) 0x7f61dc003570 con 0x7f61f006cb80 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 >> v1:172.21.15.112:6790/0 conn(0x7f61f006aa70 legacy=0x7f61f00a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 >> v1:172.21.15.112:6789/0 conn(0x7f61f00a2270 legacy=0x7f61f00a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: _finish_hunting 0 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 monclient: found mon.noname-c 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f00a0e90 con 0x7f61f006cb80 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: _finish_auth 0 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f61e0001a50 con 0x7f61f006cb80 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 5 monclient: authenticate success, global_id 17359 2024-09-18T11:46:15.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: dump: 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:15.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61e0001c70 con 0x7f61f006cb80 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 1 -- 172.21.15.112:0/2256518547 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f61e00020a0 con 0x7f61f006cb80 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 4 set_mon_vals no callback set 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61e7fff700 10 monclient: dump: 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:15.995 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 10 monclient: shutdown 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 1 -- 172.21.15.112:0/2256518547 >> v1:172.21.15.129:6789/0 conn(0x7f61f006cb80 legacy=0x7f61f0137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 1 -- 172.21.15.112:0/2256518547 shutdown_connections 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.987+0000 7f61f74ee700 1 -- 172.21.15.112:0/2256518547 >> 172.21.15.112:0/2256518547 conn(0x7f61f00680d0 msgr2=0x7f61f00a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 -- 172.21.15.112:0/2256518547 shutdown_connections 2024-09-18T11:46:15.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 -- 172.21.15.112:0/2256518547 wait complete. 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient: build_initial_monmap 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient: monmap: 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:16.000 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 librados: starting msgr at 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 librados: starting objecter 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 Processor -- start 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 -- start start 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 librados: setting wanted keys 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 librados: calling monclient init 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient: init 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:46:16.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient(hunting): picked mon.noname-c con 0x7f61f006aa70 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient(hunting): picked mon.noname-b con 0x7f61f006cb80 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient(hunting): picked mon.noname-a con 0x7f61f00a2270 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f006b2c0 con 0x7f61f00a2270 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f009d7b0 con 0x7f61f006cb80 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f01204b0 con 0x7f61f006aa70 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f74ee700 10 monclient(hunting): authenticate will time out at 947.833933s 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f528a700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f61f006aa70 0x7f61f013f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42604/0 (socket says 172.21.15.112:42604) 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f5a8b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f61f00a2270 0x7f61f0147900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38812/0 (socket says 172.21.15.112:38812) 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.991+0000 7f61f528a700 1 -- 172.21.15.112:0/453916402 learned_addr learned my addr 172.21.15.112:0/453916402 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f5a8b700 1 -- 172.21.15.112:0/453916402 >> v1:172.21.15.112:6789/0 conn(0x7f61f00a2270 legacy=0x7f61f0147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f528a700 1 -- 172.21.15.112:0/453916402 >> v1:172.21.15.112:6790/0 conn(0x7f61f006aa70 legacy=0x7f61f013f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 client.?.objecter ms_handle_connect 0x7f61f00a2270 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 client.?.objecter ms_handle_connect 0x7f61f006aa70 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 client.?.objecter ms_handle_connect 0x7f61f006cb80 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f4a89700 1 -- 172.21.15.112:0/453916402 >> v1:172.21.15.129:6789/0 conn(0x7f61f006cb80 legacy=0x7f61f0133390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15317906 0 0) 0x7f61f01204b0 con 0x7f61f006aa70 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): my global_id is 29153 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61d4004f10 con 0x7f61f006aa70 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055281720 0 0) 0x7f61f009d7b0 con 0x7f61f006cb80 2024-09-18T11:46:16.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): my global_id is 17362 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f01204b0 con 0x7f61f006cb80 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 931988659 0 0) 0x7f61d4004f10 con 0x7f61f006aa70 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f009d7b0 con 0x7f61f006aa70 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f61e8001ea0 con 0x7f61f006aa70 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): dump: 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.004 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2767929458 0 0) 0x7f61f009d7b0 con 0x7f61f006aa70 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 >> v1:172.21.15.129:6789/0 conn(0x7f61f006cb80 legacy=0x7f61f0133390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 >> v1:172.21.15.112:6789/0 conn(0x7f61f00a2270 legacy=0x7f61f0147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient: _finish_hunting 0 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 monclient: found mon.c 2024-09-18T11:46:16.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f011e980 con 0x7f61f006aa70 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient: _finish_auth 0 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 5 monclient: authenticate success, global_id 29153 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 10 monclient: _renew_subs 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f011ee30 con 0x7f61f006aa70 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 10 client.29153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 10 monclient: _renew_subs 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f0140350 con 0x7f61f006aa70 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 1 librados: init done 2024-09-18T11:46:16.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61e8004f60 con 0x7f61f006aa70 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f61e8005270 con 0x7f61f006aa70 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e6ffd700 4 set_mon_vals no callback set 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 10 monclient: dump: 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:16.007 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:16.008 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:16.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f61e80053d0 con 0x7f61f006aa70 2024-09-18T11:46:16.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:16.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:16.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.995+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d00050b0 con 0x7f61f006aa70 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.999+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6524+0+0 (unknown 1660440463 0 0) 0x7f61e8044650 con 0x7f61f006aa70 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.999+0000 7f61e57fa700 10 client.29153.objecter ms_dispatch 0x7f61f0000fd0 osd_map(70..70 src has 1..70) v4 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.999+0000 7f61e57fa700 3 client.29153.objecter handle_osd_map got epochs [70,70] > 0 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.999+0000 7f61e57fa700 3 client.29153.objecter handle_osd_map decoding full epoch 70 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.999+0000 7f61e57fa700 20 client.29153.objecter dump_active .. 0 homeless 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.999+0000 7f61f4a89700 10 monclient: get_auth_request con 0x7f61d4038280 auth_method 0 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:15.999+0000 7f61e57fa700 10 client.29153.objecter ms_handle_connect 0x7f61d4038280 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.003+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f61e8014e30 con 0x7f61f006aa70 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.003+0000 7f61e57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:16.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.003+0000 7f61e57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:16.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.235+0000 7f61f74ee700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:46:16.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.235+0000 7f61f74ee700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:16.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.235+0000 7f61f74ee700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:16.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.235+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f61d00058e0 con 0x7f61f006aa70 2024-09-18T11:46:16.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.235+0000 7f61e57fa700 1 -- 172.21.15.112:0/453916402 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3709538754) 0x7f61e8014830 con 0x7f61f006aa70 2024-09-18T11:46:16.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61e57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:16.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61e57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:16.241 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:46:16.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 10 librados: watch_flush enter 2024-09-18T11:46:16.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 10 librados: watch_flush exit 2024-09-18T11:46:16.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 20 client.29153.objecter shutdown clearing up homeless session... 2024-09-18T11:46:16.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 10 client.29153.objecter successfully canceled tick 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 >> v1:172.21.15.129:6812/20806 conn(0x7f61d4038280 legacy=0x7f61d403a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 10 monclient: shutdown 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 >> v1:172.21.15.112:6790/0 conn(0x7f61f006aa70 legacy=0x7f61f013f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 shutdown_connections 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 >> 172.21.15.112:0/453916402 conn(0x7f61f00680d0 msgr2=0x7f61f009e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 shutdown_connections 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 1 -- 172.21.15.112:0/453916402 wait complete. 2024-09-18T11:46:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.239+0000 7f61f74ee700 1 librados: shutdown 2024-09-18T11:46:16.254 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:46:16.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:46:16.254 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient: get_monmap_and_config 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient: build_initial_monmap 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient: monmap: 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:16.358892+0000 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:16.358892+0000 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.359 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:16.360 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:16.360 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:16.360 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 1 Processor -- start 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 1 -- start start 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient: init 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient(hunting): picked mon.noname-b con 0x7f5b440a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient(hunting): picked mon.noname-c con 0x7f5b44096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient(hunting): picked mon.noname-a con 0x7f5b4413ca20 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:16.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4406e910 con 0x7f5b4413ca20 2024-09-18T11:46:16.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4406e630 con 0x7f5b44096a60 2024-09-18T11:46:16.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b440a2270 con 0x7f5b440a1ea0 2024-09-18T11:46:16.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:16.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4af1a700 10 monclient(hunting): authenticate will time out at 948.199723s 2024-09-18T11:46:16.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4a719700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5b4413ca20 0x7f5b4413ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38816/0 (socket says 172.21.15.112:38816) 2024-09-18T11:46:16.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b4a719700 1 -- 172.21.15.112:0/3867207383 learned_addr learned my addr 172.21.15.112:0/3867207383 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:16.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.355+0000 7f5b49f18700 1 --1- 172.21.15.112:0/3867207383 >> v1:172.21.15.112:6790/0 conn(0x7f5b440a1ea0 0x7f5b4406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42612/0 (socket says 172.21.15.112:42612) 2024-09-18T11:46:16.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.359+0000 7f5b4a719700 1 -- 172.21.15.112:0/3867207383 >> v1:172.21.15.112:6789/0 conn(0x7f5b4413ca20 legacy=0x7f5b4413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.359+0000 7f5b49f18700 1 -- 172.21.15.112:0/3867207383 >> v1:172.21.15.112:6790/0 conn(0x7f5b440a1ea0 legacy=0x7f5b4406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.359+0000 7f5b49717700 1 -- 172.21.15.112:0/3867207383 >> v1:172.21.15.129:6789/0 conn(0x7f5b44096a60 legacy=0x7f5b4406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289839198 0 0) 0x7f5b4406e910 con 0x7f5b4413ca20 2024-09-18T11:46:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): my global_id is 74097 2024-09-18T11:46:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30003570 con 0x7f5b4413ca20 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397258456 0 0) 0x7f5b4406e630 con 0x7f5b44096a60 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): my global_id is 17365 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b4406e910 con 0x7f5b44096a60 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283075603 0 0) 0x7f5b440a2270 con 0x7f5b440a1ea0 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient(hunting): my global_id is 29156 2024-09-18T11:46:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b4406e630 con 0x7f5b440a1ea0 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541953069 0 0) 0x7f5b30003570 con 0x7f5b4413ca20 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 >> v1:172.21.15.112:6790/0 conn(0x7f5b440a1ea0 legacy=0x7f5b4406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 >> v1:172.21.15.129:6789/0 conn(0x7f5b44096a60 legacy=0x7f5b4406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient: _finish_hunting 0 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 monclient: found mon.noname-a 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4406eed0 con 0x7f5b4413ca20 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient: _finish_auth 0 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b4af1a700 5 monclient: authenticate success, global_id 74097 2024-09-18T11:46:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5b40001940 con 0x7f5b4413ca20 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b4af1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b48f16700 10 monclient: dump: 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.887+0000 7f5b4af1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b40001bd0 con 0x7f5b4413ca20 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b48f16700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b48f16700 1 -- 172.21.15.112:0/3867207383 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5b40004a90 con 0x7f5b4413ca20 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 4 set_mon_vals no callback set 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b48f16700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b48f16700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b48f16700 10 monclient: dump: 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:16.898 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 10 monclient: shutdown 2024-09-18T11:46:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 1 -- 172.21.15.112:0/3867207383 >> v1:172.21.15.112:6789/0 conn(0x7f5b4413ca20 legacy=0x7f5b4413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 1 -- 172.21.15.112:0/3867207383 shutdown_connections 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 1 -- 172.21.15.112:0/3867207383 >> 172.21.15.112:0/3867207383 conn(0x7f5b44067da0 msgr2=0x7f5b440a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 1 -- 172.21.15.112:0/3867207383 shutdown_connections 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 1 -- 172.21.15.112:0/3867207383 wait complete. 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 10 monclient: build_initial_monmap 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 10 monclient: monmap: 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 1 librados: starting msgr at 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.891+0000 7f5b4af1a700 1 librados: starting objecter 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 1 Processor -- start 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 1 -- start start 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 1 librados: setting wanted keys 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 1 librados: calling monclient init 2024-09-18T11:46:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 10 monclient: init 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 10 monclient(hunting): picked mon.noname-c con 0x7f5b44096a60 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 10 monclient(hunting): picked mon.noname-a con 0x7f5b440a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 10 monclient(hunting): picked mon.noname-b con 0x7f5b4413ca20 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4406e110 con 0x7f5b440a1ea0 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4409dd70 con 0x7f5b4413ca20 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b44125200 con 0x7f5b44096a60 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4af1a700 10 monclient(hunting): authenticate will time out at 948.736914s 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b49717700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5b440a1ea0 0x7f5b44123a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38826/0 (socket says 172.21.15.112:38826) 2024-09-18T11:46:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b49717700 1 -- 172.21.15.112:0/2775937074 learned_addr learned my addr 172.21.15.112:0/2775937074 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b49717700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6789/0 conn(0x7f5b440a1ea0 legacy=0x7f5b44123a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b39ffb700 10 client.?.objecter ms_handle_connect 0x7f5b440a1ea0 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b39ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b4a719700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.129:6789/0 conn(0x7f5b4413ca20 legacy=0x7f5b4414ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b39ffb700 10 client.?.objecter ms_handle_connect 0x7f5b4413ca20 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b39ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b49f18700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6790/0 conn(0x7f5b44096a60 legacy=0x7f5b44123340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b39ffb700 10 client.?.objecter ms_handle_connect 0x7f5b44096a60 2024-09-18T11:46:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.895+0000 7f5b39ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244895451 0 0) 0x7f5b44125200 con 0x7f5b44096a60 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient(hunting): my global_id is 29159 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b28004f10 con 0x7f5b44096a60 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1676911501 0 0) 0x7f5b28004f10 con 0x7f5b44096a60 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b44125200 con 0x7f5b44096a60 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5b34001c70 con 0x7f5b44096a60 2024-09-18T11:46:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient(hunting): dump: 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:16.907 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1927425260 0 0) 0x7f5b44125200 con 0x7f5b44096a60 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.129:6789/0 conn(0x7f5b4413ca20 legacy=0x7f5b4414ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6789/0 conn(0x7f5b440a1ea0 legacy=0x7f5b44123a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 monclient: found mon.c 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b44124cf0 con 0x7f5b44096a60 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient: _finish_auth 0 2024-09-18T11:46:16.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 5 monclient: authenticate success, global_id 29159 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 10 monclient: _renew_subs 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b441258c0 con 0x7f5b44096a60 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 10 client.29159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 10 monclient: _renew_subs 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b44124300 con 0x7f5b44096a60 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 1 librados: init done 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b34004be0 con 0x7f5b44096a60 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5b34004ef0 con 0x7f5b44096a60 2024-09-18T11:46:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 monclient: dump: 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:16.911 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b3bfff700 4 set_mon_vals no callback set 2024-09-18T11:46:16.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5b34005050 con 0x7f5b44096a60 2024-09-18T11:46:16.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b4af1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:16.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6902+0+0 (unknown 3391087911 0 0) 0x7f5b340444f0 con 0x7f5b44096a60 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 10 client.29159.objecter ms_dispatch 0x7f5b44000fd0 osd_map(71..71 src has 1..71) v4 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 3 client.29159.objecter handle_osd_map got epochs [71,71] > 0 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.899+0000 7f5b39ffb700 3 client.29159.objecter handle_osd_map decoding full epoch 71 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.903+0000 7f5b39ffb700 20 client.29159.objecter dump_active .. 0 homeless 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.903+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5b24006590 con 0x7f5b24002840 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.903+0000 7f5b49717700 10 monclient: get_auth_request con 0x7f5b28038120 auth_method 0 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.907+0000 7f5b39ffb700 10 client.29159.objecter ms_handle_connect 0x7f5b28038120 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b4a719700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6789/0 conn(0x7f5b24002840 legacy=0x7f5b24004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 10 client.29159.objecter ms_handle_connect 0x7f5b24002840 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 10 client.29159.objecter resend_mon_ops 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754347537 0 0) 0x7f5b24006590 con 0x7f5b24002840 2024-09-18T11:46:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 10 monclient(hunting): my global_id is 74100 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b44125200 con 0x7f5b24002840 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2929945805 0 0) 0x7f5b44125200 con 0x7f5b24002840 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b24006590 con 0x7f5b24002840 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5b40001360 con 0x7f5b24002840 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 10 monclient: dump: 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3414162962 0 0) 0x7f5b24006590 con 0x7f5b24002840 2024-09-18T11:46:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.915+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b2400a140 con 0x7f5b24002840 2024-09-18T11:46:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.919+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5b40004ae0 con 0x7f5b24002840 2024-09-18T11:46:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.919+0000 7f5b39ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.919+0000 7f5b39ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.919+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6789/0 conn(0x7f5b24002840 legacy=0x7f5b24004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.919+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6789/0 conn(0x7f5b24002840 legacy=0x7f5b24004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:16.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.927+0000 7f5b4af1a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:46:16.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.927+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5b24009670 con 0x7f5b240090e0 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 client.29159.objecter ms_handle_connect 0x7f5b240090e0 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 client.29159.objecter resend_mon_ops 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4a719700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6789/0 conn(0x7f5b240090e0 legacy=0x7f5b24008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130657004 0 0) 0x7f5b24009670 con 0x7f5b240090e0 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 monclient(hunting): my global_id is 74103 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b24006590 con 0x7f5b240090e0 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1990847834 0 0) 0x7f5b24006590 con 0x7f5b240090e0 2024-09-18T11:46:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b24009670 con 0x7f5b240090e0 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5b400051a0 con 0x7f5b240090e0 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 monclient: dump: 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2965214788 0 0) 0x7f5b24009670 con 0x7f5b240090e0 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5b24009930 con 0x7f5b240090e0 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 1910168179) 0x7f5b400053f0 con 0x7f5b240090e0 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6789/0 conn(0x7f5b240090e0 legacy=0x7f5b24008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b39ffb700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6789/0 conn(0x7f5b240090e0 legacy=0x7f5b24008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 14, 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:46:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:46:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:46:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:46:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:46:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:46:16.953 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:16.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 10 librados: watch_flush enter 2024-09-18T11:46:16.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 10 librados: watch_flush exit 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 20 client.29159.objecter shutdown clearing up homeless session... 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 10 client.29159.objecter successfully canceled tick 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.129:6812/20806 conn(0x7f5b28038120 legacy=0x7f5b2803a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 10 monclient: shutdown 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 >> v1:172.21.15.112:6790/0 conn(0x7f5b44096a60 legacy=0x7f5b44123340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4a719700 1 -- 172.21.15.112:0/2775937074 reap_dead start 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4a719700 1 -- 172.21.15.112:0/2775937074 reap_dead start 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 shutdown_connections 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 >> 172.21.15.112:0/2775937074 conn(0x7f5b44067da0 msgr2=0x7f5b4409fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 shutdown_connections 2024-09-18T11:46:16.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 1 -- 172.21.15.112:0/2775937074 wait complete. 2024-09-18T11:46:16.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:16.935+0000 7f5b4af1a700 1 librados: shutdown 2024-09-18T11:46:16.956 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:46:16.956 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:46:16.956 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:46:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.067+0000 7f23c05e3700 10 monclient: get_monmap_and_config 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.067+0000 7f23c05e3700 10 monclient: build_initial_monmap 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.067+0000 7f23c05e3700 10 monclient: monmap: 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:17.070680+0000 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:17.070680+0000 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:17.072 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.067+0000 7f23c05e3700 1 Processor -- start 2024-09-18T11:46:17.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.067+0000 7f23c05e3700 1 -- start start 2024-09-18T11:46:17.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.067+0000 7f23c05e3700 10 monclient: init 2024-09-18T11:46:17.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:17.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:46:17.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 10 monclient(hunting): picked mon.noname-a con 0x7f23b806add0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:17.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 10 monclient(hunting): picked mon.noname-b con 0x7f23b8132c30 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:17.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 10 monclient(hunting): picked mon.noname-c con 0x7f23b8133070 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:17.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80a17f0 con 0x7f23b806add0 2024-09-18T11:46:17.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8097010 con 0x7f23b8133070 2024-09-18T11:46:17.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80972f0 con 0x7f23b8132c30 2024-09-18T11:46:17.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:17.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23be37f700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f23b806add0 0x7f23b8132520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38856/0 (socket says 172.21.15.112:38856) 2024-09-18T11:46:17.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23be37f700 1 -- 172.21.15.112:0/2580019818 learned_addr learned my addr 172.21.15.112:0/2580019818 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:17.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23c05e3700 10 monclient(hunting): authenticate will time out at 948.913383s 2024-09-18T11:46:17.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23bdb7e700 1 --1- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6790/0 conn(0x7f23b8132c30 0x7f23b8137850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42640/0 (socket says 172.21.15.112:42640) 2024-09-18T11:46:17.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23be37f700 1 -- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6789/0 conn(0x7f23b806add0 legacy=0x7f23b8132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23bdb7e700 1 -- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6790/0 conn(0x7f23b8132c30 legacy=0x7f23b8137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23bdb7e700 1 -- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6790/0 conn(0x7f23b8132c30 legacy=0x7f23b8137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23bdb7e700 1 -- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6790/0 conn(0x7f23b8132c30 legacy=0x7f23b8137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23bdb7e700 1 --1- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6790/0 conn(0x7f23b8132c30 0x7f23b8137850 :-1 s=OPENED pgs=8777 cs=1 l=1).handle_message read tag failed 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23bdb7e700 1 --1- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6790/0 conn(0x7f23b8132c30 0x7f23b8137850 :-1 s=OPENED pgs=8777 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23bd37d700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6790/0 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.071+0000 7f23beb80700 1 -- 172.21.15.112:0/2580019818 >> v1:172.21.15.129:6789/0 conn(0x7f23b8133070 legacy=0x7f23b813af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102248092 0 0) 0x7f23b80a17f0 con 0x7f23b806add0 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient(hunting): my global_id is 74106 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a4003570 con 0x7f23b806add0 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007509779 0 0) 0x7f23a4003570 con 0x7f23b806add0 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6790/0 conn(0x7f23b8132c30 legacy=0x7f23b8137850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 >> v1:172.21.15.129:6789/0 conn(0x7f23b8133070 legacy=0x7f23b813af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: _finish_hunting 0 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 monclient: found mon.noname-a 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b80975d0 con 0x7f23b806add0 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: _finish_auth 0 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 5 monclient: authenticate success, global_id 74106 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f23b4001940 con 0x7f23b806add0 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: dump: 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.081 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b4001c10 con 0x7f23b806add0 2024-09-18T11:46:17.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:17.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:17.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 4 set_mon_vals no callback set 2024-09-18T11:46:17.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 1 -- 172.21.15.112:0/2580019818 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f23b4004a90 con 0x7f23b806add0 2024-09-18T11:46:17.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:17.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23bd37d700 10 monclient: dump: 2024-09-18T11:46:17.083 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.083 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.084 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.084 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.084 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.084 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.084 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.084 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.084 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.084 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 10 monclient: shutdown 2024-09-18T11:46:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 -- 172.21.15.112:0/2580019818 >> v1:172.21.15.112:6789/0 conn(0x7f23b806add0 legacy=0x7f23b8132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 -- 172.21.15.112:0/2580019818 shutdown_connections 2024-09-18T11:46:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 -- 172.21.15.112:0/2580019818 >> 172.21.15.112:0/2580019818 conn(0x7f23b80a5f90 msgr2=0x7f23b80a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 -- 172.21.15.112:0/2580019818 shutdown_connections 2024-09-18T11:46:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 -- 172.21.15.112:0/2580019818 wait complete. 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 10 monclient: build_initial_monmap 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 10 monclient: monmap: 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:17.092 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 librados: starting msgr at 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 librados: starting objecter 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 Processor -- start 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 -- start start 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 librados: setting wanted keys 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 1 librados: calling monclient init 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.075+0000 7f23c05e3700 10 monclient: init 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:17.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 10 monclient(hunting): picked mon.noname-a con 0x7f23b806add0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 10 monclient(hunting): picked mon.noname-b con 0x7f23b8132c30 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 10 monclient(hunting): picked mon.noname-c con 0x7f23b8133070 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b811e8b0 con 0x7f23b806add0 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b811eb60 con 0x7f23b8132c30 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b811ee40 con 0x7f23b8133070 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23c05e3700 10 monclient(hunting): authenticate will time out at 948.920755s 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23be37f700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f23b806add0 0x7f23b8131760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38872/0 (socket says 172.21.15.112:38872) 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23be37f700 1 -- 172.21.15.112:0/3254256254 learned_addr learned my addr 172.21.15.112:0/3254256254 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23be37f700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6789/0 conn(0x7f23b806add0 legacy=0x7f23b8131760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 client.?.objecter ms_handle_connect 0x7f23b806add0 2024-09-18T11:46:17.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 client.?.objecter ms_handle_connect 0x7f23b8133070 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23beb80700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6790/0 conn(0x7f23b8133070 legacy=0x7f23b814a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23bdb7e700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.129:6789/0 conn(0x7f23b8132c30 legacy=0x7f23b8146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 client.?.objecter ms_handle_connect 0x7f23b8132c30 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013692685 0 0) 0x7f23b811ee40 con 0x7f23b8133070 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): my global_id is 29162 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f239c004f10 con 0x7f23b8133070 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676797469 0 0) 0x7f23b811e8b0 con 0x7f23b806add0 2024-09-18T11:46:17.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): my global_id is 74109 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b811ee40 con 0x7f23b806add0 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278748980 0 0) 0x7f23b811eb60 con 0x7f23b8132c30 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 10 monclient(hunting): my global_id is 17371 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.079+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b811e8b0 con 0x7f23b8132c30 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 968341178 0 0) 0x7f239c004f10 con 0x7f23b8133070 2024-09-18T11:46:17.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b811eb60 con 0x7f23b8133070 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f23b0001f40 con 0x7f23b8133070 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): dump: 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1263917774 0 0) 0x7f23b811ee40 con 0x7f23b806add0 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f239c004f10 con 0x7f23b806add0 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f23b4004830 con 0x7f23b806add0 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): dump: 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046478441 0 0) 0x7f23b811e8b0 con 0x7f23b8132c30 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b811ee40 con 0x7f23b8132c30 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f23a8001be0 con 0x7f23b8132c30 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient(hunting): dump: 2024-09-18T11:46:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1415820254 0 0) 0x7f23b811eb60 con 0x7f23b8133070 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.129:6789/0 conn(0x7f23b8132c30 legacy=0x7f23b8146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6789/0 conn(0x7f23b806add0 legacy=0x7f23b8131760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 monclient: found mon.c 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b811f120 con 0x7f23b8133070 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient: _finish_auth 0 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 5 monclient: authenticate success, global_id 29162 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 10 monclient: _renew_subs 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b8131e70 con 0x7f23b8133070 2024-09-18T11:46:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 10 client.29162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 10 monclient: _renew_subs 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b811db40 con 0x7f23b8133070 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 1 librados: init done 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b0002530 con 0x7f23b8133070 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f23b0002900 con 0x7f23b8133070 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23bcb7c700 4 set_mon_vals no callback set 2024-09-18T11:46:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 monclient: dump: 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f23b0002ae0 con 0x7f23b8133070 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f23a0006590 con 0x7f23a0002840 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6902+0+0 (unknown 3391087911 0 0) 0x7f23b00447b0 con 0x7f23b8133070 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 10 client.29162.objecter ms_dispatch 0x7f23b8000fd0 osd_map(71..71 src has 1..71) v4 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 3 client.29162.objecter handle_osd_map got epochs [71,71] > 0 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 3 client.29162.objecter handle_osd_map decoding full epoch 71 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23ae7fc700 20 client.29162.objecter dump_active .. 0 homeless 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.083+0000 7f23bdb7e700 10 monclient: get_auth_request con 0x7f239c038670 auth_method 0 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 client.29162.objecter ms_handle_connect 0x7f23a0002840 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 client.29162.objecter resend_mon_ops 2024-09-18T11:46:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23be37f700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6790/0 conn(0x7f23a0002840 legacy=0x7f23a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793418861 0 0) 0x7f23a0006590 con 0x7f23a0002840 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 monclient(hunting): my global_id is 29165 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b811eb60 con 0x7f23a0002840 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4286664510 0 0) 0x7f23b811eb60 con 0x7f23a0002840 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a0006590 con 0x7f23a0002840 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f23b4005210 con 0x7f23a0002840 2024-09-18T11:46:17.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 monclient: dump: 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 243399728 0 0) 0x7f23a0006590 con 0x7f23a0002840 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23a000a140 con 0x7f23a0002840 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 client.29162.objecter ms_handle_connect 0x7f239c038670 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f23b40053f0 con 0x7f23a0002840 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6790/0 conn(0x7f23a0002840 legacy=0x7f23a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.087+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6790/0 conn(0x7f23a0002840 legacy=0x7f23a0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.099+0000 7f23c05e3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:46:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.099+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f23a0009670 con 0x7f23a00090e0 2024-09-18T11:46:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.099+0000 7f23ae7fc700 10 client.29162.objecter ms_handle_connect 0x7f23a00090e0 2024-09-18T11:46:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.099+0000 7f23be37f700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6790/0 conn(0x7f23a00090e0 legacy=0x7f23a0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.099+0000 7f23ae7fc700 10 client.29162.objecter resend_mon_ops 2024-09-18T11:46:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524000265 0 0) 0x7f23a0009670 con 0x7f23a00090e0 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 10 monclient(hunting): my global_id is 29168 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a0006590 con 0x7f23a00090e0 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4262097858 0 0) 0x7f23a0006590 con 0x7f23a00090e0 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a0009670 con 0x7f23a00090e0 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f23b4005a70 con 0x7f23a00090e0 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 10 monclient: dump: 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3234320511 0 0) 0x7f23a0009670 con 0x7f23a00090e0 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f23a0009930 con 0x7f23a00090e0 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 1305979731) 0x7f23b4006fb0 con 0x7f23a00090e0 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6790/0 conn(0x7f23a00090e0 legacy=0x7f23a0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23ae7fc700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6790/0 conn(0x7f23a00090e0 legacy=0x7f23a0008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 14, 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.112 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 1, 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:17.113 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:46:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:17.116 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.117 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.119 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:17.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.121 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.122 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 6 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.123 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:46:17.124 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 10 librados: watch_flush enter 2024-09-18T11:46:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 10 librados: watch_flush exit 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 20 client.29162.objecter shutdown clearing up homeless session... 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 10 client.29162.objecter successfully canceled tick 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.129:6812/20806 conn(0x7f239c038670 legacy=0x7f239c03ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 10 monclient: shutdown 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 >> v1:172.21.15.112:6790/0 conn(0x7f23b8133070 legacy=0x7f23b814a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23beb80700 1 -- 172.21.15.112:0/3254256254 reap_dead start 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23beb80700 1 -- 172.21.15.112:0/3254256254 reap_dead start 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 shutdown_connections 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 >> 172.21.15.112:0/3254256254 conn(0x7f23b80a5f90 msgr2=0x7f23b80a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 shutdown_connections 2024-09-18T11:46:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 1 -- 172.21.15.112:0/3254256254 wait complete. 2024-09-18T11:46:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.103+0000 7f23c05e3700 1 librados: shutdown 2024-09-18T11:46:17.127 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:46:17.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:46:17.127 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:46:17.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient: get_monmap_and_config 2024-09-18T11:46:17.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient: build_initial_monmap 2024-09-18T11:46:17.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient: monmap: 2024-09-18T11:46:17.242 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:17.242 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.242 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:17.242017+0000 2024-09-18T11:46:17.242 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:17.242017+0000 2024-09-18T11:46:17.243 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:17.243 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.243 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:17.243 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:17.243 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:17.243 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 1 Processor -- start 2024-09-18T11:46:17.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 1 -- start start 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient: init 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fbc200a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fbc200a0750 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fbc201385b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc20097270 con 0x7fbc200a0380 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc200a3170 con 0x7fbc201385b0 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc20096cb0 con 0x7fbc200a0750 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1f59e700 10 monclient(hunting): authenticate will time out at 949.083760s 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1e59c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbc200a0380 0x7fbc20097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38878/0 (socket says 172.21.15.112:38878) 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1e59c700 1 -- 172.21.15.112:0/665654417 learned_addr learned my addr 172.21.15.112:0/665654417 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.239+0000 7fbc1dd9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fbc200a0750 0x7fbc200a0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42676/0 (socket says 172.21.15.112:42676) 2024-09-18T11:46:17.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1e59c700 1 -- 172.21.15.112:0/665654417 >> v1:172.21.15.112:6789/0 conn(0x7fbc200a0380 legacy=0x7fbc20097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1dd9b700 1 -- 172.21.15.112:0/665654417 >> v1:172.21.15.112:6790/0 conn(0x7fbc200a0750 legacy=0x7fbc200a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1ed9d700 1 -- 172.21.15.112:0/665654417 >> v1:172.21.15.129:6789/0 conn(0x7fbc201385b0 legacy=0x7fbc2013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141563131 0 0) 0x7fbc20097270 con 0x7fbc200a0380 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): my global_id is 74112 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c003570 con 0x7fbc200a0380 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914336653 0 0) 0x7fbc20096cb0 con 0x7fbc200a0750 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): my global_id is 29171 2024-09-18T11:46:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc20097270 con 0x7fbc200a0750 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254074807 0 0) 0x7fbc200a3170 con 0x7fbc201385b0 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient(hunting): my global_id is 17374 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc20096cb0 con 0x7fbc201385b0 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003197164 0 0) 0x7fbc0c003570 con 0x7fbc200a0380 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 >> v1:172.21.15.112:6790/0 conn(0x7fbc200a0750 legacy=0x7fbc200a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 >> v1:172.21.15.129:6789/0 conn(0x7fbc201385b0 legacy=0x7fbc2013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: _finish_hunting 0 2024-09-18T11:46:17.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 monclient: found mon.noname-a 2024-09-18T11:46:17.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:17.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:46:17.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200a1420 con 0x7fbc200a0380 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: _finish_auth 0 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbc10001cc0 con 0x7fbc200a0380 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: dump: 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1f59e700 5 monclient: authenticate success, global_id 74112 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc10001f50 con 0x7fbc200a0380 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 1 -- 172.21.15.112:0/665654417 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbc10004e10 con 0x7fbc200a0380 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1d59a700 10 monclient: dump: 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1f59e700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:17.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.243+0000 7fbc1f59e700 4 set_mon_vals no callback set 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient: shutdown 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- 172.21.15.112:0/665654417 >> v1:172.21.15.112:6789/0 conn(0x7fbc200a0380 legacy=0x7fbc20097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- 172.21.15.112:0/665654417 shutdown_connections 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- 172.21.15.112:0/665654417 >> 172.21.15.112:0/665654417 conn(0x7fbc200a5990 msgr2=0x7fbc200a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- 172.21.15.112:0/665654417 shutdown_connections 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- 172.21.15.112:0/665654417 wait complete. 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient: build_initial_monmap 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient: monmap: 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 librados: starting msgr at 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 librados: starting objecter 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 Processor -- start 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- start start 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 librados: setting wanted keys 2024-09-18T11:46:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 librados: calling monclient init 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient: init 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fbc200a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fbc200a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fbc201385b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc20138cb0 con 0x7fbc200a0750 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc200a2010 con 0x7fbc201385b0 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2011e200 con 0x7fbc200a0380 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1f59e700 10 monclient(hunting): authenticate will time out at 949.090726s 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1dd9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbc200a0750 0x7fbc20146690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38886/0 (socket says 172.21.15.112:38886) 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1e59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fbc200a0380 0x7fbc2011c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42682/0 (socket says 172.21.15.112:42682) 2024-09-18T11:46:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.247+0000 7fbc1dd9b700 1 -- 172.21.15.112:0/2615627310 learned_addr learned my addr 172.21.15.112:0/2615627310 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1dd9b700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.112:6789/0 conn(0x7fbc200a0750 legacy=0x7fbc20146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 client.?.objecter ms_handle_connect 0x7fbc200a0750 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 client.?.objecter ms_handle_connect 0x7fbc200a0380 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1e59c700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.112:6790/0 conn(0x7fbc200a0380 legacy=0x7fbc2011c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293707053 0 0) 0x7fbc20138cb0 con 0x7fbc200a0750 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): my global_id is 74115 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc00004f10 con 0x7fbc200a0750 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 client.?.objecter ms_handle_connect 0x7fbc201385b0 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1ed9d700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6789/0 conn(0x7fbc201385b0 legacy=0x7fbc20149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150531765 0 0) 0x7fbc2011e200 con 0x7fbc200a0380 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): my global_id is 29174 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc20138cb0 con 0x7fbc200a0380 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1682021754 0 0) 0x7fbc00004f10 con 0x7fbc200a0750 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc2011e200 con 0x7fbc200a0750 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625756223 0 0) 0x7fbc200a2010 con 0x7fbc201385b0 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): my global_id is 17377 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc00004f10 con 0x7fbc201385b0 2024-09-18T11:46:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbc08004830 con 0x7fbc200a0750 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): dump: 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.264 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 563676619 0 0) 0x7fbc20138cb0 con 0x7fbc200a0380 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc200a2010 con 0x7fbc200a0380 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbc10004bb0 con 0x7fbc200a0380 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient(hunting): dump: 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3746650879 0 0) 0x7fbc2011e200 con 0x7fbc200a0750 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.112:6790/0 conn(0x7fbc200a0380 legacy=0x7fbc2011c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6789/0 conn(0x7fbc201385b0 legacy=0x7fbc20149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient: _finish_hunting 0 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 monclient: found mon.a 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2011da90 con 0x7fbc200a0750 2024-09-18T11:46:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 10 monclient: _finish_auth 0 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 5 monclient: authenticate success, global_id 74115 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 10 monclient: _renew_subs 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2011dce0 con 0x7fbc200a0750 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 10 client.74115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 10 monclient: _renew_subs 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2009e0a0 con 0x7fbc200a0750 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 1 librados: init done 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.251+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbc2005f2e0 con 0x7fbc2011ccd0 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc08004d30 con 0x7fbc200a0750 2024-09-18T11:46:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbc080019f0 con 0x7fbc200a0750 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc1cd99700 4 set_mon_vals no callback set 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient: dump: 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fbc08004f60 con 0x7fbc200a0750 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6902+0+0 (unknown 3391087911 0 0) 0x7fbc08044450 con 0x7fbc200a0750 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 client.74115.objecter ms_dispatch 0x7fbc20000fd0 osd_map(71..71 src has 1..71) v4 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 3 client.74115.objecter handle_osd_map got epochs [71,71] > 0 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 3 client.74115.objecter handle_osd_map decoding full epoch 71 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc1ed9d700 10 monclient: get_auth_request con 0x7fbc000385b0 auth_method 0 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 20 client.74115.objecter dump_active .. 0 homeless 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc1e59c700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6789/0 conn(0x7fbc2011ccd0 legacy=0x7fbc20099980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 client.74115.objecter ms_handle_connect 0x7fbc2011ccd0 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 client.74115.objecter resend_mon_ops 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257541694 0 0) 0x7fbc2005f2e0 con 0x7fbc2011ccd0 2024-09-18T11:46:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient(hunting): my global_id is 17380 2024-09-18T11:46:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2011e200 con 0x7fbc2011ccd0 2024-09-18T11:46:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2102220830 0 0) 0x7fbc2011e200 con 0x7fbc2011ccd0 2024-09-18T11:46:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc2005f2e0 con 0x7fbc2011ccd0 2024-09-18T11:46:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbc10005200 con 0x7fbc2011ccd0 2024-09-18T11:46:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 monclient: dump: 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1005696302 0 0) 0x7fbc2005f2e0 con 0x7fbc2011ccd0 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc2011eea0 con 0x7fbc2011ccd0 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.255+0000 7fbc167fc700 10 client.74115.objecter ms_handle_connect 0x7fbc000385b0 2024-09-18T11:46:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.259+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fbc10007440 con 0x7fbc2011ccd0 2024-09-18T11:46:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.259+0000 7fbc167fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.259+0000 7fbc167fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.259+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6789/0 conn(0x7fbc2011ccd0 legacy=0x7fbc20099980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.259+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6789/0 conn(0x7fbc2011ccd0 legacy=0x7fbc20099980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.267+0000 7fbc1f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:46:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.267+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbc20137f60 con 0x7fbc20137b90 2024-09-18T11:46:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc1e59c700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6789/0 conn(0x7fbc20137b90 legacy=0x7fbc2002ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 client.74115.objecter ms_handle_connect 0x7fbc20137b90 2024-09-18T11:46:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 client.74115.objecter resend_mon_ops 2024-09-18T11:46:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999114584 0 0) 0x7fbc20137f60 con 0x7fbc20137b90 2024-09-18T11:46:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 monclient(hunting): my global_id is 17383 2024-09-18T11:46:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2005f2e0 con 0x7fbc20137b90 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2622066404 0 0) 0x7fbc2005f2e0 con 0x7fbc20137b90 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc20137f60 con 0x7fbc20137b90 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbc100058e0 con 0x7fbc20137b90 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 monclient: dump: 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.276 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 600528303 0 0) 0x7fbc20137f60 con 0x7fbc20137b90 2024-09-18T11:46:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbc200635d0 con 0x7fbc20137b90 2024-09-18T11:46:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 3845748614) 0x7fbc10006c70 con 0x7fbc20137b90 2024-09-18T11:46:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:46:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6789/0 conn(0x7fbc20137b90 legacy=0x7fbc2002ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc167fc700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6789/0 conn(0x7fbc20137b90 legacy=0x7fbc2002ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:17.277 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 14, 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 1, 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:17.278 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:46:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:46:17.281 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:46:17.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:46:17.283 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:46:17.284 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:17.285 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:17.286 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:46:17.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 7 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:46:17.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:17.289 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:17.289 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:17.289 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:17.289 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:17.289 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:17.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:17.289 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:46:17.289 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc1f59e700 10 librados: watch_flush enter 2024-09-18T11:46:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc1f59e700 10 librados: watch_flush exit 2024-09-18T11:46:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc1f59e700 20 client.74115.objecter shutdown clearing up homeless session... 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.271+0000 7fbc1f59e700 10 client.74115.objecter successfully canceled tick 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.129:6812/20806 conn(0x7fbc000385b0 legacy=0x7fbc0003aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 10 monclient: shutdown 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 >> v1:172.21.15.112:6789/0 conn(0x7fbc200a0750 legacy=0x7fbc20146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1ed9d700 1 -- 172.21.15.112:0/2615627310 reap_dead start 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1ed9d700 1 -- 172.21.15.112:0/2615627310 reap_dead start 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 shutdown_connections 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 >> 172.21.15.112:0/2615627310 conn(0x7fbc200a5990 msgr2=0x7fbc2009ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 shutdown_connections 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 1 -- 172.21.15.112:0/2615627310 wait complete. 2024-09-18T11:46:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.275+0000 7fbc1f59e700 1 librados: shutdown 2024-09-18T11:46:17.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:46:17.293 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T11:46:17.293 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T11:46:17.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.403+0000 7f916f59e700 10 monclient: get_monmap_and_config 2024-09-18T11:46:17.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.403+0000 7f916f59e700 10 monclient: build_initial_monmap 2024-09-18T11:46:17.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.403+0000 7f916f59e700 10 monclient: monmap: 2024-09-18T11:46:17.407 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:17.407182+0000 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:17.407182+0000 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:17.408 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.403+0000 7f916f59e700 1 Processor -- start 2024-09-18T11:46:17.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.403+0000 7f916f59e700 1 -- start start 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.403+0000 7f916f59e700 10 monclient: init 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f91700a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f917012edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f917012f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91700981a0 con 0x7f917012edc0 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917006c120 con 0x7f91700a2040 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917006eef0 con 0x7f917012f8a0 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916f59e700 10 monclient(hunting): authenticate will time out at 949.248959s 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916dd9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f917012edc0 0x7f917012f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38896/0 (socket says 172.21.15.112:38896) 2024-09-18T11:46:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916dd9b700 1 -- 172.21.15.112:0/3516478800 learned_addr learned my addr 172.21.15.112:0/3516478800 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916dd9b700 1 -- 172.21.15.112:0/3516478800 >> v1:172.21.15.112:6789/0 conn(0x7f917012edc0 legacy=0x7f917012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916ed9d700 1 -- 172.21.15.112:0/3516478800 >> v1:172.21.15.112:6790/0 conn(0x7f917012f8a0 legacy=0x7f9170137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916e59c700 1 -- 172.21.15.112:0/3516478800 >> v1:172.21.15.129:6789/0 conn(0x7f91700a2040 legacy=0x7f9170097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932933444 0 0) 0x7f917006eef0 con 0x7f917012f8a0 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 10 monclient(hunting): my global_id is 29177 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9150003570 con 0x7f917012f8a0 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083315981 0 0) 0x7f91700981a0 con 0x7f917012edc0 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 10 monclient(hunting): my global_id is 74118 2024-09-18T11:46:17.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.407+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f917006eef0 con 0x7f917012edc0 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417470809 0 0) 0x7f917006c120 con 0x7f91700a2040 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient(hunting): my global_id is 17386 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91700981a0 con 0x7f91700a2040 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066426847 0 0) 0x7f9150003570 con 0x7f917012f8a0 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 >> v1:172.21.15.129:6789/0 conn(0x7f91700a2040 legacy=0x7f9170097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 >> v1:172.21.15.112:6789/0 conn(0x7f917012edc0 legacy=0x7f917012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: _finish_hunting 0 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 monclient: found mon.noname-b 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9170096dd0 con 0x7f917012f8a0 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: _finish_auth 0 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9160001dd0 con 0x7f917012f8a0 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 5 monclient: authenticate success, global_id 29177 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: dump: 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9160001f30 con 0x7f917012f8a0 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 4 set_mon_vals no callback set 2024-09-18T11:46:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 1 -- 172.21.15.112:0/3516478800 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9160002420 con 0x7f917012f8a0 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916d59a700 10 monclient: dump: 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.418 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 10 monclient: shutdown 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 -- 172.21.15.112:0/3516478800 >> v1:172.21.15.112:6790/0 conn(0x7f917012f8a0 legacy=0x7f9170137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 -- 172.21.15.112:0/3516478800 shutdown_connections 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 -- 172.21.15.112:0/3516478800 >> 172.21.15.112:0/3516478800 conn(0x7f9170067da0 msgr2=0x7f91700a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 -- 172.21.15.112:0/3516478800 shutdown_connections 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 -- 172.21.15.112:0/3516478800 wait complete. 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 10 monclient: build_initial_monmap 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 10 monclient: monmap: 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:17.420 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.420 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:17.420 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:17.420 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:17.420 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 librados: starting msgr at 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 librados: starting objecter 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 Processor -- start 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 -- start start 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 librados: setting wanted keys 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 1 librados: calling monclient init 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.411+0000 7f916f59e700 10 monclient: init 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f91700a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f917012edc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f917012f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91700a0160 con 0x7f91700a2040 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917006d2e0 con 0x7f917012f8a0 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9170135b70 con 0x7f917012edc0 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient(hunting): authenticate will time out at 949.256247s 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916dd9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f917012edc0 0x7f9170135460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42692/0 (socket says 172.21.15.112:42692) 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916dd9b700 1 -- 172.21.15.112:0/2956583665 learned_addr learned my addr 172.21.15.112:0/2956583665 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916dd9b700 1 -- 172.21.15.112:0/2956583665 >> v1:172.21.15.112:6790/0 conn(0x7f917012edc0 legacy=0x7f9170135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 client.?.objecter ms_handle_connect 0x7f917012edc0 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 client.?.objecter ms_handle_connect 0x7f91700a2040 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916e59c700 1 -- 172.21.15.112:0/2956583665 >> v1:172.21.15.112:6789/0 conn(0x7f91700a2040 legacy=0x7f9170134d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716048027 0 0) 0x7f9170135b70 con 0x7f917012edc0 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): my global_id is 29180 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9154004f10 con 0x7f917012edc0 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916ed9d700 1 -- 172.21.15.112:0/2956583665 >> v1:172.21.15.129:6789/0 conn(0x7f917012f8a0 legacy=0x7f9170147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 client.?.objecter ms_handle_connect 0x7f917012f8a0 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839199407 0 0) 0x7f91700a0160 con 0x7f91700a2040 2024-09-18T11:46:17.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): my global_id is 74121 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9170135b70 con 0x7f91700a2040 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2041125647 0 0) 0x7f9154004f10 con 0x7f917012edc0 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91700a0160 con 0x7f917012edc0 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f91580019a0 con 0x7f917012edc0 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): dump: 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277394107 0 0) 0x7f917006d2e0 con 0x7f917012f8a0 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient(hunting): my global_id is 17389 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9154004f10 con 0x7f917012f8a0 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2147029695 0 0) 0x7f91700a0160 con 0x7f917012edc0 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 >> v1:172.21.15.129:6789/0 conn(0x7f917012f8a0 legacy=0x7f9170147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 >> v1:172.21.15.112:6789/0 conn(0x7f91700a2040 legacy=0x7f9170134d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient: _finish_hunting 0 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 monclient: found mon.c 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9170138450 con 0x7f917012edc0 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient: _finish_auth 0 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 5 monclient: authenticate success, global_id 29180 2024-09-18T11:46:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient: _renew_subs 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9170137180 con 0x7f917012edc0 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 client.29180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient: _renew_subs 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91701363c0 con 0x7f917012edc0 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f916f59e700 1 librados: init done 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9158004c20 con 0x7f917012edc0 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.415+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9158004d90 con 0x7f917012edc0 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f916cd99700 4 set_mon_vals no callback set 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 10 monclient: dump: 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f915800f690 con 0x7f917012edc0 2024-09-18T11:46:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f916f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f916f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f916f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91480050b0 con 0x7f917012edc0 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6902+0+0 (unknown 3391087911 0 0) 0x7f9158044340 con 0x7f917012edc0 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 10 client.29180.objecter ms_dispatch 0x7f9170000fd0 osd_map(71..71 src has 1..71) v4 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 3 client.29180.objecter handle_osd_map got epochs [71,71] > 0 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 3 client.29180.objecter handle_osd_map decoding full epoch 71 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f916e59c700 10 monclient: get_auth_request con 0x7f91540383a0 auth_method 0 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.419+0000 7f91667fc700 20 client.29180.objecter dump_active .. 0 homeless 2024-09-18T11:46:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.423+0000 7f91667fc700 10 client.29180.objecter ms_handle_connect 0x7f91540383a0 2024-09-18T11:46:17.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.423+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f9158014930 con 0x7f917012edc0 2024-09-18T11:46:17.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.423+0000 7f91667fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:17.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.423+0000 7f91667fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:17.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.643+0000 7f916f59e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T11:46:17.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.643+0000 7f916f59e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T11:46:17.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.643+0000 7f916f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:17.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.643+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f9148005b70 con 0x7f917012edc0 2024-09-18T11:46:17.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f91667fc700 1 -- 172.21.15.112:0/2956583665 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f9158028320 con 0x7f917012edc0 2024-09-18T11:46:17.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f91667fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T11:46:17.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f91667fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:17.651 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.651 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=7,logm=65} crc {auth=2357799396,config=3140514009,health=2867994915,kv=2459836832,logm=2422652215}) 2024-09-18T11:46:17.651 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=11,kv=8,logm=68} crc {auth=3115817956,config=3140514009,health=2977030664,kv=1310296871,logm=3716956857}) 2024-09-18T11:46:17.651 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.651 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=7,logm=65} crc {auth=2357799396,config=3140514009,health=2867994915,kv=2459836832,logm=2422652215}) 2024-09-18T11:46:17.651 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.647+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=1,health=12,kv=7,logm=68} crc {auth=4200867177,config=1228700967,health=2867994915,kv=1310296871,logm=2770535753}) 2024-09-18T11:46:17.653 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.653 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643051599}) 2024-09-18T11:46:17.653 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1692503831}) 2024-09-18T11:46:17.653 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.653 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643051599}) 2024-09-18T11:46:17.653 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4048273000}) 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 10 librados: watch_flush enter 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 10 librados: watch_flush exit 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 20 client.29180.objecter shutdown clearing up homeless session... 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 10 client.29180.objecter successfully canceled tick 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 >> v1:172.21.15.129:6812/20806 conn(0x7f91540383a0 legacy=0x7f915403a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 10 monclient: shutdown 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 >> v1:172.21.15.112:6790/0 conn(0x7f917012edc0 legacy=0x7f9170135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 shutdown_connections 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 >> 172.21.15.112:0/2956583665 conn(0x7f9170067da0 msgr2=0x7f917006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 shutdown_connections 2024-09-18T11:46:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 1 -- 172.21.15.112:0/2956583665 wait complete. 2024-09-18T11:46:17.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f916f59e700 1 librados: shutdown 2024-09-18T11:46:17.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109838978}) 2024-09-18T11:46:17.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1988510642}) 2024-09-18T11:46:17.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109838978}) 2024-09-18T11:46:17.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.651+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2709795292}) 2024-09-18T11:46:17.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1106215914}) 2024-09-18T11:46:17.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2539648933}) 2024-09-18T11:46:17.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1106215914}) 2024-09-18T11:46:17.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3711692869}) 2024-09-18T11:46:17.660 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.660 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250760470}) 2024-09-18T11:46:17.660 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190798204}) 2024-09-18T11:46:17.660 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.660 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250760470}) 2024-09-18T11:46:17.660 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.655+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653352381}) 2024-09-18T11:46:17.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276373827}) 2024-09-18T11:46:17.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3273146655}) 2024-09-18T11:46:17.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276373827}) 2024-09-18T11:46:17.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1590201303}) 2024-09-18T11:46:17.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503089874}) 2024-09-18T11:46:17.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3933667670}) 2024-09-18T11:46:17.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503089874}) 2024-09-18T11:46:17.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3541719359}) 2024-09-18T11:46:17.663 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.663 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91462875}) 2024-09-18T11:46:17.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262297099}) 2024-09-18T11:46:17.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91462875}) 2024-09-18T11:46:17.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.659+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=123980797}) 2024-09-18T11:46:17.665 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.665 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643872729}) 2024-09-18T11:46:17.665 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3341181497}) 2024-09-18T11:46:17.665 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.665 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643872729}) 2024-09-18T11:46:17.665 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1713269846}) 2024-09-18T11:46:17.666 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.667 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441809655}) 2024-09-18T11:46:17.667 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071697592}) 2024-09-18T11:46:17.667 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.667 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441809655}) 2024-09-18T11:46:17.667 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1905095201}) 2024-09-18T11:46:17.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T11:46:17.668 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.669 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.663+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=4018390620,mdsmap=1531151339,mgr=3190381165,mgr_command_descs=47663136,mgr_metadata=454364145,mgrstat=2528896726,mon_config_key=3035149395,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=842112476,osd_snap=3718248}) 2024-09-18T11:46:17.669 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=3} crc {logm=4005862223,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=518688800,mon_config_key=868353578,monmap=3140514009,osd_metadata=998566107}) 2024-09-18T11:46:17.669 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.669 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=4018390620,mdsmap=1531151339,mgr=3190381165,mgr_command_descs=47663136,mgr_metadata=454364145,mgrstat=2528896726,mon_config_key=3035149395,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=842112476,osd_snap=3718248}) 2024-09-18T11:46:17.669 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=2,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=4} crc {logm=2275994172,mdsmap=3140514009,mgr=3183642700,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=3056176584,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=842112476,osd_snap=3215490119}) 2024-09-18T11:46:17.670 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.670 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=3155026058,osdmap=4188578943}) 2024-09-18T11:46:17.670 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3098950065,osdmap=3207084282}) 2024-09-18T11:46:17.670 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.670 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=3155026058,osdmap=4188578943}) 2024-09-18T11:46:17.671 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.667+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=2891054416,osdmap=2265163685}) 2024-09-18T11:46:17.675 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.671+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.675 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.671+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=965366729}) 2024-09-18T11:46:17.675 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.671+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=525766377}) 2024-09-18T11:46:17.675 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.671+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:46:17.675 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.671+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=965366729}) 2024-09-18T11:46:17.675 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:46:17.671+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=32} crc {osdmap=3077063504}) 2024-09-18T11:46:18.669 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:46:18.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:46:18.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.775+0000 7fa9c0cac700 10 monclient: get_monmap_and_config 2024-09-18T11:46:18.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.775+0000 7fa9c0cac700 10 monclient: build_initial_monmap 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.775+0000 7fa9c0cac700 10 monclient: monmap: 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:18.779402+0000 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:18.779402+0000 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:18.780 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:18.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 1 Processor -- start 2024-09-18T11:46:18.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 1 -- start start 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 10 monclient: init 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 10 monclient(hunting): picked mon.noname-b con 0x7fa9bc0a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 10 monclient(hunting): picked mon.noname-c con 0x7fa9bc06e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 10 monclient(hunting): picked mon.noname-a con 0x7fa9bc135300 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc0a1600 con 0x7fa9bc135300 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc06eda0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc06b5a0 con 0x7fa9bc0a1230 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9c0cac700 10 monclient(hunting): authenticate will time out at 950.621041s 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9ba59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa9bc0a1230 0x7fa9bc06a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42694/0 (socket says 172.21.15.112:42694) 2024-09-18T11:46:18.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9ba59c700 1 -- 172.21.15.112:0/4170500671 learned_addr learned my addr 172.21.15.112:0/4170500671 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:18.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9ba59c700 1 -- 172.21.15.112:0/4170500671 >> v1:172.21.15.112:6790/0 conn(0x7fa9bc0a1230 legacy=0x7fa9bc06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:18.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9bad9d700 1 -- 172.21.15.112:0/4170500671 >> v1:172.21.15.112:6789/0 conn(0x7fa9bc135300 legacy=0x7fa9bc1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:18.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9b9d9b700 1 -- 172.21.15.112:0/4170500671 >> v1:172.21.15.129:6789/0 conn(0x7fa9bc06e9d0 legacy=0x7fa9bc06ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:18.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236100711 0 0) 0x7fa9bc06b5a0 con 0x7fa9bc0a1230 2024-09-18T11:46:18.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9b959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:18.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9b959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:18.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9b959a700 10 monclient(hunting): my global_id is 29189 2024-09-18T11:46:18.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.779+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a4003570 con 0x7fa9bc0a1230 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005092323 0 0) 0x7fa9bc0a1600 con 0x7fa9bc135300 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient(hunting): my global_id is 74130 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc06b5a0 con 0x7fa9bc135300 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482612634 0 0) 0x7fa9bc06eda0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient(hunting): my global_id is 17398 2024-09-18T11:46:18.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc0a1600 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3322394053 0 0) 0x7fa9a4003570 con 0x7fa9bc0a1230 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 >> v1:172.21.15.129:6789/0 conn(0x7fa9bc06e9d0 legacy=0x7fa9bc06ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 >> v1:172.21.15.112:6789/0 conn(0x7fa9bc135300 legacy=0x7fa9bc1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: _finish_hunting 0 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 monclient: found mon.noname-b 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc09f9e0 con 0x7fa9bc0a1230 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: _finish_auth 0 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa9ac001980 con 0x7fa9bc0a1230 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 5 monclient: authenticate success, global_id 29189 2024-09-18T11:46:18.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: dump: 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:18.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9ac001ae0 con 0x7fa9bc0a1230 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 1 -- 172.21.15.112:0/4170500671 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa9ac0048f0 con 0x7fa9bc0a1230 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 4 set_mon_vals no callback set 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b959a700 10 monclient: dump: 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:18.790 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient: shutdown 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- 172.21.15.112:0/4170500671 >> v1:172.21.15.112:6790/0 conn(0x7fa9bc0a1230 legacy=0x7fa9bc06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- 172.21.15.112:0/4170500671 shutdown_connections 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- 172.21.15.112:0/4170500671 >> 172.21.15.112:0/4170500671 conn(0x7fa9bc0680d0 msgr2=0x7fa9bc0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- 172.21.15.112:0/4170500671 shutdown_connections 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- 172.21.15.112:0/4170500671 wait complete. 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient: build_initial_monmap 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient: monmap: 2024-09-18T11:46:18.791 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 librados: starting msgr at 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 librados: starting objecter 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 Processor -- start 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- start start 2024-09-18T11:46:18.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 librados: setting wanted keys 2024-09-18T11:46:18.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 librados: calling monclient init 2024-09-18T11:46:18.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient: init 2024-09-18T11:46:18.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:18.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:46:18.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient(hunting): picked mon.noname-a con 0x7fa9bc06e9d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:18.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient(hunting): picked mon.noname-c con 0x7fa9bc0a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:18.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient(hunting): picked mon.noname-b con 0x7fa9bc135300 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:18.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc12ccc0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc12e2d0 con 0x7fa9bc135300 2024-09-18T11:46:18.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc06ef90 con 0x7fa9bc0a1230 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9c0cac700 10 monclient(hunting): authenticate will time out at 950.627677s 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9ba59c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa9bc06e9d0 0x7fa9bc1308f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38918/0 (socket says 172.21.15.112:38918) 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9ba59c700 1 -- 172.21.15.112:0/2491380728 learned_addr learned my addr 172.21.15.112:0/2491380728 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.783+0000 7fa9b9d9b700 1 --1- 172.21.15.112:0/2491380728 >> v1:172.21.15.112:6790/0 conn(0x7fa9bc0a1230 0x7fa9bc131000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42704/0 (socket says 172.21.15.112:42704) 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9ba59c700 1 -- 172.21.15.112:0/2491380728 >> v1:172.21.15.112:6789/0 conn(0x7fa9bc06e9d0 legacy=0x7fa9bc1308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 client.?.objecter ms_handle_connect 0x7fa9bc06e9d0 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 client.?.objecter ms_handle_connect 0x7fa9bc0a1230 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9b9d9b700 1 -- 172.21.15.112:0/2491380728 >> v1:172.21.15.112:6790/0 conn(0x7fa9bc0a1230 legacy=0x7fa9bc131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 client.?.objecter ms_handle_connect 0x7fa9bc135300 2024-09-18T11:46:18.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9bad9d700 1 -- 172.21.15.112:0/2491380728 >> v1:172.21.15.129:6789/0 conn(0x7fa9bc135300 legacy=0x7fa9bc14a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315250809 0 0) 0x7fa9bc12ccc0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): my global_id is 74133 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa99c004f10 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190468864 0 0) 0x7fa9bc06ef90 con 0x7fa9bc0a1230 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): my global_id is 29192 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc12ccc0 con 0x7fa9bc0a1230 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756212555 0 0) 0x7fa9bc12e2d0 con 0x7fa9bc135300 2024-09-18T11:46:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): my global_id is 17401 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc06ef90 con 0x7fa9bc135300 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 343108060 0 0) 0x7fa99c004f10 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9bc12e2d0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3229481571 0 0) 0x7fa9bc12ccc0 con 0x7fa9bc0a1230 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa99c004f10 con 0x7fa9bc0a1230 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa9ac001c50 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): dump: 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:18.799 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa9b0001c40 con 0x7fa9bc0a1230 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:18.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient(hunting): dump: 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3610843939 0 0) 0x7fa9bc12e2d0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 >> v1:172.21.15.112:6790/0 conn(0x7fa9bc0a1230 legacy=0x7fa9bc131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 >> v1:172.21.15.129:6789/0 conn(0x7fa9bc135300 legacy=0x7fa9bc14a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:18.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 monclient: found mon.a 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc131710 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient: _finish_auth 0 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 5 monclient: authenticate success, global_id 74133 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 10 monclient: _renew_subs 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9bc132990 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 10 client.74133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 10 monclient: _renew_subs 2024-09-18T11:46:18.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9bc133050 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9c0cac700 1 librados: init done 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9ac004f50 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa9ac005260 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9b8d99700 4 set_mon_vals no callback set 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.787+0000 7fa9aa7fc700 10 monclient: dump: 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:18.803 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9c0cac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9c0cac700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9c0cac700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a00050b0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa9ac00fae0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6526+0+0 (unknown 3855670250 0 0) 0x7fa9ac0444c0 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9aa7fc700 10 client.74133.objecter ms_dispatch 0x7fa9bc000fd0 osd_map(72..72 src has 1..72) v4 2024-09-18T11:46:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9aa7fc700 3 client.74133.objecter handle_osd_map got epochs [72,72] > 0 2024-09-18T11:46:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9aa7fc700 3 client.74133.objecter handle_osd_map decoding full epoch 72 2024-09-18T11:46:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9aa7fc700 20 client.74133.objecter dump_active .. 0 homeless 2024-09-18T11:46:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.791+0000 7fa9b9d9b700 10 monclient: get_auth_request con 0x7fa99c038480 auth_method 0 2024-09-18T11:46:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.795+0000 7fa9aa7fc700 10 client.74133.objecter ms_handle_connect 0x7fa99c038480 2024-09-18T11:46:18.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.799+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fa9ac014c20 con 0x7fa9bc06e9d0 2024-09-18T11:46:18.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.799+0000 7fa9aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:18.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:18.799+0000 7fa9aa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:19.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:46:19.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:19.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa9a00058e0 con 0x7fa9bc06e9d0 2024-09-18T11:46:19.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9aa7fc700 1 -- 172.21.15.112:0/2491380728 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1628449522) 0x7fa9ac014620 con 0x7fa9bc06e9d0 2024-09-18T11:46:19.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:19.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9aa7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:19.042 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:46:19.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 10 librados: watch_flush enter 2024-09-18T11:46:19.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 10 librados: watch_flush exit 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 20 client.74133.objecter shutdown clearing up homeless session... 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 10 client.74133.objecter successfully canceled tick 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.039+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 >> v1:172.21.15.129:6812/20806 conn(0x7fa99c038480 legacy=0x7fa99c03a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.043+0000 7fa9c0cac700 10 monclient: shutdown 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.043+0000 7fa9c0cac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.043+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 >> v1:172.21.15.112:6789/0 conn(0x7fa9bc06e9d0 legacy=0x7fa9bc1308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.043+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 shutdown_connections 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.043+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 >> 172.21.15.112:0/2491380728 conn(0x7fa9bc0680d0 msgr2=0x7fa9bc09e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.043+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 shutdown_connections 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.043+0000 7fa9c0cac700 1 -- 172.21.15.112:0/2491380728 wait complete. 2024-09-18T11:46:19.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.043+0000 7fa9c0cac700 1 librados: shutdown 2024-09-18T11:46:19.056 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:46:19.056 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T11:46:19.056 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:46:19.056 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:46:19.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.159+0000 7fc4bd409700 10 monclient: get_monmap_and_config 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.159+0000 7fc4bd409700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.159+0000 7fc4bd409700 10 monclient: monmap: 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:19.163583+0000 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:19.163583+0000 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:19.164 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.159+0000 7fc4bd409700 1 Processor -- start 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.159+0000 7fc4bd409700 1 -- start start 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.159+0000 7fc4bd409700 10 monclient: init 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 10 monclient(hunting): picked mon.noname-a con 0x7fc4b806df70 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 10 monclient(hunting): picked mon.noname-b con 0x7fc4b806ea50 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 10 monclient(hunting): picked mon.noname-c con 0x7fc4b806d970 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b806b960 con 0x7fc4b806df70 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b806cac0 con 0x7fc4b806d970 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b806c7e0 con 0x7fc4b806ea50 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4bd409700 10 monclient(hunting): authenticate will time out at 951.004721s 2024-09-18T11:46:19.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b67fc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc4b806ea50 0x7fc4b81342f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34320/0 (socket says 172.21.15.112:34320) 2024-09-18T11:46:19.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b6ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc4b806df70 0x7fc4b806e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47090/0 (socket says 172.21.15.112:47090) 2024-09-18T11:46:19.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b67fc700 1 -- 172.21.15.112:0/3026885135 learned_addr learned my addr 172.21.15.112:0/3026885135 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b6ffd700 1 -- 172.21.15.112:0/3026885135 >> v1:172.21.15.112:6789/0 conn(0x7fc4b806df70 legacy=0x7fc4b806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b67fc700 1 -- 172.21.15.112:0/3026885135 >> v1:172.21.15.112:6790/0 conn(0x7fc4b806ea50 legacy=0x7fc4b81342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b77fe700 1 -- 172.21.15.112:0/3026885135 >> v1:172.21.15.129:6789/0 conn(0x7fc4b806d970 legacy=0x7fc4b8137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699954362 0 0) 0x7fc4b806b960 con 0x7fc4b806df70 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): my global_id is 74142 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a4003570 con 0x7fc4b806df70 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112132081 0 0) 0x7fc4b806c7e0 con 0x7fc4b806ea50 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): my global_id is 29201 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b806b960 con 0x7fc4b806ea50 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791395175 0 0) 0x7fc4b806cac0 con 0x7fc4b806d970 2024-09-18T11:46:19.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 10 monclient(hunting): my global_id is 17410 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.163+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b806c7e0 con 0x7fc4b806d970 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 820873340 0 0) 0x7fc4a4003570 con 0x7fc4b806df70 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 >> v1:172.21.15.112:6790/0 conn(0x7fc4b806ea50 legacy=0x7fc4b81342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 >> v1:172.21.15.129:6789/0 conn(0x7fc4b806d970 legacy=0x7fc4b8137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 1 monclient: found mon.noname-a 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b806cda0 con 0x7fc4b806df70 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 5 monclient: authenticate success, global_id 74142 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc4a0001a00 con 0x7fc4b806df70 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: dump: 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.172 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4a0001be0 con 0x7fc4b806df70 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 4 set_mon_vals no callback set 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 1 -- 172.21.15.112:0/3026885135 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc4a00049d0 con 0x7fc4b806df70 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4b5ffb700 10 monclient: dump: 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.173 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 10 monclient: shutdown 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 -- 172.21.15.112:0/3026885135 >> v1:172.21.15.112:6789/0 conn(0x7fc4b806df70 legacy=0x7fc4b806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 -- 172.21.15.112:0/3026885135 shutdown_connections 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 -- 172.21.15.112:0/3026885135 >> 172.21.15.112:0/3026885135 conn(0x7fc4b80680d0 msgr2=0x7fc4b80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 -- 172.21.15.112:0/3026885135 shutdown_connections 2024-09-18T11:46:19.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 -- 172.21.15.112:0/3026885135 wait complete. 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 10 monclient: monmap: 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:19.179 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 librados: starting msgr at 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 librados: starting objecter 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 Processor -- start 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 -- start start 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 librados: setting wanted keys 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 1 librados: calling monclient init 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.167+0000 7fc4bd409700 10 monclient: init 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 10 monclient(hunting): picked mon.noname-b con 0x7fc4b806d970 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 10 monclient(hunting): picked mon.noname-c con 0x7fc4b806df70 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 10 monclient(hunting): picked mon.noname-a con 0x7fc4b806ea50 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b811e5b0 con 0x7fc4b806ea50 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b811d3b0 con 0x7fc4b806d970 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b813e190 con 0x7fc4b806df70 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4bd409700 10 monclient(hunting): authenticate will time out at 951.012740s 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4b77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc4b806ea50 0x7fc4b8145d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47106/0 (socket says 172.21.15.112:47106) 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4b77fe700 1 -- 172.21.15.112:0/584923890 learned_addr learned my addr 172.21.15.112:0/584923890 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4b67fc700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6790/0 conn(0x7fc4b806df70 legacy=0x7fc4b81425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4b77fe700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6789/0 conn(0x7fc4b806ea50 legacy=0x7fc4b8145d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 client.?.objecter ms_handle_connect 0x7fc4b806df70 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 client.?.objecter ms_handle_connect 0x7fc4b806ea50 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4b6ffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.129:6789/0 conn(0x7fc4b806d970 legacy=0x7fc4b8130170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 client.?.objecter ms_handle_connect 0x7fc4b806d970 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73650449 0 0) 0x7fc4b813e190 con 0x7fc4b806df70 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): my global_id is 29204 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc498004f10 con 0x7fc4b806df70 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062150737 0 0) 0x7fc4b811e5b0 con 0x7fc4b806ea50 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): my global_id is 74145 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b813e190 con 0x7fc4b806ea50 2024-09-18T11:46:19.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907604088 0 0) 0x7fc4b811d3b0 con 0x7fc4b806d970 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): my global_id is 17413 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b811e5b0 con 0x7fc4b806d970 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2758331233 0 0) 0x7fc498004f10 con 0x7fc4b806df70 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4b811d3b0 con 0x7fc4b806df70 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc4b0001cb0 con 0x7fc4b806df70 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): dump: 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.183 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4107319533 0 0) 0x7fc4b813e190 con 0x7fc4b806ea50 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc498004f10 con 0x7fc4b806ea50 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc4a8002030 con 0x7fc4b806ea50 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.171+0000 7fc4aeffd700 10 monclient(hunting): dump: 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.184 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2013364073 0 0) 0x7fc4b811d3b0 con 0x7fc4b806df70 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.129:6789/0 conn(0x7fc4b806d970 legacy=0x7fc4b8130170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6789/0 conn(0x7fc4b806ea50 legacy=0x7fc4b8145d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 monclient: found mon.c 2024-09-18T11:46:19.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b811e2a0 con 0x7fc4b806df70 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 5 monclient: authenticate success, global_id 29204 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 10 monclient: _renew_subs 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b811e8c0 con 0x7fc4b806df70 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 10 client.29204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 10 monclient: _renew_subs 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b811bc40 con 0x7fc4b806df70 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 1 librados: init done 2024-09-18T11:46:19.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b0004be0 con 0x7fc4b806df70 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc4b0004ef0 con 0x7fc4b806df70 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4b57fa700 4 set_mon_vals no callback set 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient: dump: 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.187 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fc4b0005050 con 0x7fc4b806df70 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc49c006590 con 0x7fc49c002840 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7fc4b0044340 con 0x7fc4b806df70 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 client.29204.objecter ms_dispatch 0x7fc4b8000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 3 client.29204.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 3 client.29204.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 20 client.29204.objecter dump_active .. 0 homeless 2024-09-18T11:46:19.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4b77fe700 10 monclient: get_auth_request con 0x7fc498038500 auth_method 0 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 client.29204.objecter ms_handle_connect 0x7fc49c002840 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 client.29204.objecter resend_mon_ops 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4b6ffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6789/0 conn(0x7fc49c002840 legacy=0x7fc49c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577731989 0 0) 0x7fc49c006590 con 0x7fc49c002840 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 10 monclient(hunting): my global_id is 74148 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.175+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b811d3b0 con 0x7fc49c002840 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1596284183 0 0) 0x7fc4b811d3b0 con 0x7fc49c002840 2024-09-18T11:46:19.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc49c006590 con 0x7fc49c002840 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc4a0004f20 con 0x7fc49c002840 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 10 monclient: dump: 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.191 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3042621926 0 0) 0x7fc49c006590 con 0x7fc49c002840 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc49c00a140 con 0x7fc49c002840 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 10 client.29204.objecter ms_handle_connect 0x7fc498038500 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fc4a0005080 con 0x7fc49c002840 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6789/0 conn(0x7fc49c002840 legacy=0x7fc49c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.179+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6789/0 conn(0x7fc49c002840 legacy=0x7fc49c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:19.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.187+0000 7fc4bd409700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:46:19.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.187+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc49c009670 con 0x7fc49c0090e0 2024-09-18T11:46:19.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4b6ffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6789/0 conn(0x7fc49c0090e0 legacy=0x7fc49c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 client.29204.objecter ms_handle_connect 0x7fc49c0090e0 2024-09-18T11:46:19.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 client.29204.objecter resend_mon_ops 2024-09-18T11:46:19.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555910175 0 0) 0x7fc49c009670 con 0x7fc49c0090e0 2024-09-18T11:46:19.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 monclient(hunting): my global_id is 74151 2024-09-18T11:46:19.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c006590 con 0x7fc49c0090e0 2024-09-18T11:46:19.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 508642934 0 0) 0x7fc49c006590 con 0x7fc49c0090e0 2024-09-18T11:46:19.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc49c009670 con 0x7fc49c0090e0 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc4a00057e0 con 0x7fc49c0090e0 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 monclient: dump: 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.196 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3412337048 0 0) 0x7fc49c009670 con 0x7fc49c0090e0 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc49c009930 con 0x7fc49c0090e0 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 2815614445) 0x7fc4a0005a00 con 0x7fc49c0090e0 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6789/0 conn(0x7fc49c0090e0 legacy=0x7fc49c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4aeffd700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6789/0 conn(0x7fc49c0090e0 legacy=0x7fc49c008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:19.197 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 14, 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.198 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.199 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:19.200 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.201 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:19.202 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.203 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.204 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:46:19.205 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:46:19.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.207 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:46:19.208 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:19.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 10 librados: watch_flush enter 2024-09-18T11:46:19.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 10 librados: watch_flush exit 2024-09-18T11:46:19.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 20 client.29204.objecter shutdown clearing up homeless session... 2024-09-18T11:46:19.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 10 client.29204.objecter successfully canceled tick 2024-09-18T11:46:19.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.129:6812/20806 conn(0x7fc498038500 legacy=0x7fc49803a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 10 monclient: shutdown 2024-09-18T11:46:19.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 >> v1:172.21.15.112:6790/0 conn(0x7fc4b806df70 legacy=0x7fc4b81425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4b77fe700 1 -- 172.21.15.112:0/584923890 reap_dead start 2024-09-18T11:46:19.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 shutdown_connections 2024-09-18T11:46:19.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 >> 172.21.15.112:0/584923890 conn(0x7fc4b80680d0 msgr2=0x7fc4b809fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.191+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 shutdown_connections 2024-09-18T11:46:19.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.195+0000 7fc4bd409700 1 -- 172.21.15.112:0/584923890 wait complete. 2024-09-18T11:46:19.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.195+0000 7fc4bd409700 1 librados: shutdown 2024-09-18T11:46:19.210 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:46:19.210 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:46:19.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:46:19.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.315+0000 7fe22c92d700 10 monclient: get_monmap_and_config 2024-09-18T11:46:19.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.315+0000 7fe22c92d700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.315+0000 7fe22c92d700 10 monclient: monmap: 2024-09-18T11:46:19.319 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.319 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.319 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:19.319113+0000 2024-09-18T11:46:19.320 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:19.319113+0000 2024-09-18T11:46:19.320 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.320 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.320 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.320 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:19.320 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:19.320 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.315+0000 7fe22c92d700 1 Processor -- start 2024-09-18T11:46:19.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.315+0000 7fe22c92d700 1 -- start start 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.315+0000 7fe22c92d700 10 monclient: init 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 10 monclient(hunting): picked mon.noname-b con 0x7fe2280a2270 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 10 monclient(hunting): picked mon.noname-a con 0x7fe22806aa70 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 10 monclient(hunting): picked mon.noname-c con 0x7fe22806cb80 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22806a830 con 0x7fe22806aa70 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2280a2640 con 0x7fe22806cb80 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22806d000 con 0x7fe2280a2270 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22c92d700 10 monclient(hunting): authenticate will time out at 951.160428s 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe2280a2270 0x7fe2280a0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34328/0 (socket says 172.21.15.112:34328) 2024-09-18T11:46:19.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22659c700 1 -- 172.21.15.112:0/1087270273 learned_addr learned my addr 172.21.15.112:0/1087270273 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe225d9b700 1 -- 172.21.15.112:0/1087270273 >> v1:172.21.15.112:6789/0 conn(0x7fe22806aa70 legacy=0x7fe2280a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22659c700 1 -- 172.21.15.112:0/1087270273 >> v1:172.21.15.112:6790/0 conn(0x7fe2280a2270 legacy=0x7fe2280a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe226d9d700 1 -- 172.21.15.112:0/1087270273 >> v1:172.21.15.129:6789/0 conn(0x7fe22806cb80 legacy=0x7fe228137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984359726 0 0) 0x7fe22806d000 con 0x7fe2280a2270 2024-09-18T11:46:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): my global_id is 29207 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe210003570 con 0x7fe2280a2270 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610854690 0 0) 0x7fe22806a830 con 0x7fe22806aa70 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): my global_id is 74154 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe22806d000 con 0x7fe22806aa70 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614141174 0 0) 0x7fe2280a2640 con 0x7fe22806cb80 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient(hunting): my global_id is 17416 2024-09-18T11:46:19.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe22806a830 con 0x7fe22806cb80 2024-09-18T11:46:19.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589588046 0 0) 0x7fe22806a830 con 0x7fe22806cb80 2024-09-18T11:46:19.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 >> v1:172.21.15.112:6790/0 conn(0x7fe2280a2270 legacy=0x7fe2280a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 >> v1:172.21.15.112:6789/0 conn(0x7fe22806aa70 legacy=0x7fe2280a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 monclient: found mon.noname-c 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2280a0e90 con 0x7fe22806cb80 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe220001cb0 con 0x7fe22806cb80 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.319+0000 7fe22559a700 10 monclient: dump: 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 5 monclient: authenticate success, global_id 17416 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe220001ec0 con 0x7fe22806cb80 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22559a700 1 -- 172.21.15.112:0/1087270273 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe220002280 con 0x7fe22806cb80 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 4 set_mon_vals no callback set 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:19.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22559a700 10 monclient: dump: 2024-09-18T11:46:19.331 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.331 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.331 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.331 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.331 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.331 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.332 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.332 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.332 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.332 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 10 monclient: shutdown 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 -- 172.21.15.112:0/1087270273 >> v1:172.21.15.129:6789/0 conn(0x7fe22806cb80 legacy=0x7fe228137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 -- 172.21.15.112:0/1087270273 shutdown_connections 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 -- 172.21.15.112:0/1087270273 >> 172.21.15.112:0/1087270273 conn(0x7fe2280680d0 msgr2=0x7fe2280a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 -- 172.21.15.112:0/1087270273 shutdown_connections 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 -- 172.21.15.112:0/1087270273 wait complete. 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 10 monclient: monmap: 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 librados: starting msgr at 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 librados: starting objecter 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 Processor -- start 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 -- start start 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 librados: setting wanted keys 2024-09-18T11:46:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 1 librados: calling monclient init 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.323+0000 7fe22c92d700 10 monclient: init 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 10 monclient(hunting): picked mon.noname-a con 0x7fe22806aa70 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 10 monclient(hunting): picked mon.noname-c con 0x7fe22806cb80 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 10 monclient(hunting): picked mon.noname-b con 0x7fe2280a2270 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22811e620 con 0x7fe22806aa70 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2281399b0 con 0x7fe2280a2270 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22813a6e0 con 0x7fe22806cb80 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22c92d700 10 monclient(hunting): authenticate will time out at 951.168538s 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe22806aa70 0x7fe2281392a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47140/0 (socket says 172.21.15.112:47140) 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22659c700 1 -- 172.21.15.112:0/2930826048 learned_addr learned my addr 172.21.15.112:0/2930826048 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe225d9b700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6790/0 conn(0x7fe22806cb80 legacy=0x7fe2281424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 client.?.objecter ms_handle_connect 0x7fe22806cb80 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe22659c700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6789/0 conn(0x7fe22806aa70 legacy=0x7fe2281392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 client.?.objecter ms_handle_connect 0x7fe22806aa70 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe226d9d700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.129:6789/0 conn(0x7fe2280a2270 legacy=0x7fe228145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957722198 0 0) 0x7fe22813a6e0 con 0x7fe22806cb80 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): my global_id is 29210 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe208004f10 con 0x7fe22806cb80 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 client.?.objecter ms_handle_connect 0x7fe2280a2270 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648360327 0 0) 0x7fe22811e620 con 0x7fe22806aa70 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): my global_id is 74157 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe22813a6e0 con 0x7fe22806aa70 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647228181 0 0) 0x7fe2281399b0 con 0x7fe2280a2270 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): my global_id is 17419 2024-09-18T11:46:19.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe22811e620 con 0x7fe2280a2270 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1044046456 0 0) 0x7fe208004f10 con 0x7fe22806cb80 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2281399b0 con 0x7fe22806cb80 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe2180019c0 con 0x7fe22806cb80 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): dump: 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.345 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.327+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3891940842 0 0) 0x7fe22813a6e0 con 0x7fe22806aa70 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe208004f10 con 0x7fe22806aa70 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe21c001d40 con 0x7fe22806aa70 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient(hunting): dump: 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.346 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1173712814 0 0) 0x7fe2281399b0 con 0x7fe22806cb80 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.129:6789/0 conn(0x7fe2280a2270 legacy=0x7fe228145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6789/0 conn(0x7fe22806aa70 legacy=0x7fe2281392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 monclient: found mon.c 2024-09-18T11:46:19.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22811e340 con 0x7fe22806cb80 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 5 monclient: authenticate success, global_id 29210 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 10 monclient: _renew_subs 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22811e900 con 0x7fe22806cb80 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 10 client.29210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 10 monclient: _renew_subs 2024-09-18T11:46:19.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22811d610 con 0x7fe22806cb80 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 1 librados: init done 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2180021b0 con 0x7fe22806cb80 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe218002500 con 0x7fe22806cb80 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe224d99700 4 set_mon_vals no callback set 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 monclient: dump: 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe20c006590 con 0x7fe20c002840 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe218002660 con 0x7fe22806cb80 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7fe2180442a0 con 0x7fe22806cb80 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 client.29210.objecter ms_dispatch 0x7fe228000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 3 client.29210.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 3 client.29210.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 20 client.29210.objecter dump_active .. 0 homeless 2024-09-18T11:46:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe226d9d700 10 monclient: get_auth_request con 0x7fe208038280 auth_method 0 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 client.29210.objecter ms_handle_connect 0x7fe20c002840 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe2167fc700 10 client.29210.objecter resend_mon_ops 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.331+0000 7fe22659c700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6790/0 conn(0x7fe20c002840 legacy=0x7fe20c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290465775 0 0) 0x7fe20c006590 con 0x7fe20c002840 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 monclient(hunting): my global_id is 29213 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2281399b0 con 0x7fe20c002840 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2969735469 0 0) 0x7fe2281399b0 con 0x7fe20c002840 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe20c006590 con 0x7fe20c002840 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe21c004c80 con 0x7fe20c002840 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 monclient: dump: 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 895233704 0 0) 0x7fe20c006590 con 0x7fe20c002840 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe20c00a140 con 0x7fe20c002840 2024-09-18T11:46:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 client.29210.objecter ms_handle_connect 0x7fe208038280 2024-09-18T11:46:19.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe21c005050 con 0x7fe20c002840 2024-09-18T11:46:19.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:19.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6790/0 conn(0x7fe20c002840 legacy=0x7fe20c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.335+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6790/0 conn(0x7fe20c002840 legacy=0x7fe20c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:19.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe22c92d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe20c009670 con 0x7fe20c0090e0 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 10 client.29210.objecter ms_handle_connect 0x7fe20c0090e0 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 10 client.29210.objecter resend_mon_ops 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe22659c700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6790/0 conn(0x7fe20c0090e0 legacy=0x7fe20c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754283061 0 0) 0x7fe20c009670 con 0x7fe20c0090e0 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 10 monclient(hunting): my global_id is 29216 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c006590 con 0x7fe20c0090e0 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3157118928 0 0) 0x7fe20c006590 con 0x7fe20c0090e0 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe20c009670 con 0x7fe20c0090e0 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe21c005690 con 0x7fe20c0090e0 2024-09-18T11:46:19.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 10 monclient: dump: 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 444216944 0 0) 0x7fe20c009670 con 0x7fe20c0090e0 2024-09-18T11:46:19.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.347+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe20c009930 con 0x7fe20c0090e0 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 3872925930) 0x7fe21c0069d0 con 0x7fe20c0090e0 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe2167fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe2167fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6790/0 conn(0x7fe20c0090e0 legacy=0x7fe20c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 14, 2024-09-18T11:46:19.357 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.358 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.359 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:46:19.360 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.361 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:19.362 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:46:19.363 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:46:19.364 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.365 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.366 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.367 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 6 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:46:19.368 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe2167fc700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6790/0 conn(0x7fe20c0090e0 legacy=0x7fe20c008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 10 librados: watch_flush enter 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 10 librados: watch_flush exit 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 20 client.29210.objecter shutdown clearing up homeless session... 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 10 client.29210.objecter successfully canceled tick 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.129:6812/20806 conn(0x7fe208038280 legacy=0x7fe20803a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 10 monclient: shutdown 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 >> v1:172.21.15.112:6790/0 conn(0x7fe22806cb80 legacy=0x7fe2281424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe226d9d700 1 -- 172.21.15.112:0/2930826048 reap_dead start 2024-09-18T11:46:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe226d9d700 1 -- 172.21.15.112:0/2930826048 reap_dead start 2024-09-18T11:46:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 shutdown_connections 2024-09-18T11:46:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 >> 172.21.15.112:0/2930826048 conn(0x7fe2280680d0 msgr2=0x7fe22809f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 shutdown_connections 2024-09-18T11:46:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 1 -- 172.21.15.112:0/2930826048 wait complete. 2024-09-18T11:46:19.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.351+0000 7fe22c92d700 1 librados: shutdown 2024-09-18T11:46:19.371 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:46:19.371 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:46:19.371 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient: get_monmap_and_config 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient: monmap: 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:19.484599+0000 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:19.484599+0000 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:19.486 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 1 Processor -- start 2024-09-18T11:46:19.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 1 -- start start 2024-09-18T11:46:19.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient: init 2024-09-18T11:46:19.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:46:19.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient(hunting): picked mon.noname-a con 0x7f136406a8b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient(hunting): picked mon.noname-c con 0x7f1364132330 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient(hunting): picked mon.noname-b con 0x7f1364132e40 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1364096ad0 con 0x7f136406a8b0 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640a20e0 con 0x7f1364132330 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640a1740 con 0x7f1364132e40 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136baf0700 10 monclient(hunting): authenticate will time out at 951.326078s 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136a08d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1364132e40 0x7f136413a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34348/0 (socket says 172.21.15.112:34348) 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136a08d700 1 -- 172.21.15.112:0/953331499 learned_addr learned my addr 172.21.15.112:0/953331499 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.483+0000 7f136988c700 1 --1- 172.21.15.112:0/953331499 >> v1:172.21.15.112:6789/0 conn(0x7f136406a8b0 0x7f1364097b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47146/0 (socket says 172.21.15.112:47146) 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136a08d700 1 -- 172.21.15.112:0/953331499 >> v1:172.21.15.112:6790/0 conn(0x7f1364132e40 legacy=0x7f136413a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136988c700 1 -- 172.21.15.112:0/953331499 >> v1:172.21.15.112:6789/0 conn(0x7f136406a8b0 legacy=0x7f1364097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136908b700 1 -- 172.21.15.112:0/953331499 >> v1:172.21.15.129:6789/0 conn(0x7f1364132330 legacy=0x7f1364132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558012342 0 0) 0x7f13640a1740 con 0x7f1364132e40 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): my global_id is 29219 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1350003570 con 0x7f1364132e40 2024-09-18T11:46:19.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700450650 0 0) 0x7f1364096ad0 con 0x7f136406a8b0 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): my global_id is 74160 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13640a1740 con 0x7f136406a8b0 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354958413 0 0) 0x7f13640a20e0 con 0x7f1364132330 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient(hunting): my global_id is 17422 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1364096ad0 con 0x7f1364132330 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3286672761 0 0) 0x7f1350003570 con 0x7f1364132e40 2024-09-18T11:46:19.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 >> v1:172.21.15.129:6789/0 conn(0x7f1364132330 legacy=0x7f1364132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 >> v1:172.21.15.112:6789/0 conn(0x7f136406a8b0 legacy=0x7f1364097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 monclient: found mon.noname-b 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136413ece0 con 0x7f1364132e40 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 5 monclient: authenticate success, global_id 29219 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f135c001d80 con 0x7f1364132e40 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: dump: 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.494 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f135c001f60 con 0x7f1364132e40 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 4 set_mon_vals no callback set 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 1 -- 172.21.15.112:0/953331499 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f135c004e10 con 0x7f1364132e40 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136888a700 10 monclient: dump: 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.495 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient: shutdown 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- 172.21.15.112:0/953331499 >> v1:172.21.15.112:6790/0 conn(0x7f1364132e40 legacy=0x7f136413a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- 172.21.15.112:0/953331499 shutdown_connections 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- 172.21.15.112:0/953331499 >> 172.21.15.112:0/953331499 conn(0x7f13640a5b70 msgr2=0x7f13640a5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- 172.21.15.112:0/953331499 shutdown_connections 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- 172.21.15.112:0/953331499 wait complete. 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient: monmap: 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:19.498 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:19.498 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 librados: starting msgr at 2024-09-18T11:46:19.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 librados: starting objecter 2024-09-18T11:46:19.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 Processor -- start 2024-09-18T11:46:19.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- start start 2024-09-18T11:46:19.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 librados: setting wanted keys 2024-09-18T11:46:19.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 librados: calling monclient init 2024-09-18T11:46:19.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient: init 2024-09-18T11:46:19.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient(hunting): picked mon.noname-a con 0x7f136406a8b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient(hunting): picked mon.noname-b con 0x7f1364132330 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient(hunting): picked mon.noname-c con 0x7f1364132e40 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1364133b10 con 0x7f136406a8b0 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1364134200 con 0x7f1364132330 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640a22e0 con 0x7f1364132e40 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136baf0700 10 monclient(hunting): authenticate will time out at 951.331806s 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136a08d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1364132e40 0x7f136409e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34364/0 (socket says 172.21.15.112:34364) 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136988c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f136406a8b0 0x7f136409da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47148/0 (socket says 172.21.15.112:47148) 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.487+0000 7f136a08d700 1 -- 172.21.15.112:0/1465748618 learned_addr learned my addr 172.21.15.112:0/1465748618 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136988c700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.112:6789/0 conn(0x7f136406a8b0 legacy=0x7f136409da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 client.?.objecter ms_handle_connect 0x7f136406a8b0 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 client.?.objecter ms_handle_connect 0x7f1364132e40 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136a08d700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.112:6790/0 conn(0x7f1364132e40 legacy=0x7f136409e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126553816 0 0) 0x7f13640a22e0 con 0x7f1364132e40 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): my global_id is 29222 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1348004f10 con 0x7f1364132e40 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046534951 0 0) 0x7f1364133b10 con 0x7f136406a8b0 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): my global_id is 74163 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13640a22e0 con 0x7f136406a8b0 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 client.?.objecter ms_handle_connect 0x7f1364132330 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136908b700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6789/0 conn(0x7f1364132330 legacy=0x7f136409e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546674680 0 0) 0x7f1348004f10 con 0x7f1364132e40 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1364133b10 con 0x7f1364132e40 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f135c004bb0 con 0x7f1364132e40 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): dump: 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.506 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9161027 0 0) 0x7f1364134200 con 0x7f1364132330 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): my global_id is 17425 2024-09-18T11:46:19.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1348004f10 con 0x7f1364132330 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3962966147 0 0) 0x7f13640a22e0 con 0x7f136406a8b0 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1364134200 con 0x7f136406a8b0 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1360001bf0 con 0x7f136406a8b0 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient(hunting): dump: 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.508 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1260389658 0 0) 0x7f1364133b10 con 0x7f1364132e40 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6789/0 conn(0x7f1364132330 legacy=0x7f136409e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.112:6789/0 conn(0x7f136406a8b0 legacy=0x7f136409da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 monclient: found mon.c 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640a4410 con 0x7f1364132e40 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 5 monclient: authenticate success, global_id 29222 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 10 monclient: _renew_subs 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1364133590 con 0x7f1364132e40 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 10 client.29222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 10 monclient: _renew_subs 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f136413b040 con 0x7f1364132e40 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f136baf0700 1 librados: init done 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f135c004d70 con 0x7f1364132e40 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f135c005590 con 0x7f1364132e40 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1357fff700 4 set_mon_vals no callback set 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.491+0000 7f1355ffb700 10 monclient: dump: 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f136baf0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1344006590 con 0x7f1344002840 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f135c00fe10 con 0x7f1364132e40 2024-09-18T11:46:19.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7f135c0447b0 con 0x7f1364132e40 2024-09-18T11:46:19.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 10 client.29222.objecter ms_dispatch 0x7f1364000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:19.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 3 client.29222.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:19.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 3 client.29222.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:19.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 20 client.29222.objecter dump_active .. 0 homeless 2024-09-18T11:46:19.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f136908b700 10 monclient: get_auth_request con 0x7f1348038750 auth_method 0 2024-09-18T11:46:19.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f136988c700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6789/0 conn(0x7f1344002840 legacy=0x7f1344004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 10 client.29222.objecter ms_handle_connect 0x7f1344002840 2024-09-18T11:46:19.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 10 client.29222.objecter resend_mon_ops 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53067185 0 0) 0x7f1344006590 con 0x7f1344002840 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 10 monclient(hunting): my global_id is 17428 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.495+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1364133b10 con 0x7f1344002840 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 10 client.29222.objecter ms_handle_connect 0x7f1348038750 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4142858672 0 0) 0x7f1364133b10 con 0x7f1344002840 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1344006590 con 0x7f1344002840 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1360004f70 con 0x7f1344002840 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 10 monclient: dump: 2024-09-18T11:46:19.513 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1827947332 0 0) 0x7f1344006590 con 0x7f1344002840 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f134400a140 con 0x7f1344002840 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f13600050d0 con 0x7f1344002840 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:19.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6789/0 conn(0x7f1344002840 legacy=0x7f1344004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.499+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6789/0 conn(0x7f1344002840 legacy=0x7f1344004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:19.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.511+0000 7f136baf0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:46:19.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.511+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1344009670 con 0x7f13440090e0 2024-09-18T11:46:19.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.511+0000 7f136988c700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6789/0 conn(0x7f13440090e0 legacy=0x7f1344008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.511+0000 7f1355ffb700 10 client.29222.objecter ms_handle_connect 0x7f13440090e0 2024-09-18T11:46:19.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.511+0000 7f1355ffb700 10 client.29222.objecter resend_mon_ops 2024-09-18T11:46:19.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744167917 0 0) 0x7f1344009670 con 0x7f13440090e0 2024-09-18T11:46:19.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 10 monclient(hunting): my global_id is 17431 2024-09-18T11:46:19.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1344006590 con 0x7f13440090e0 2024-09-18T11:46:19.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 127667116 0 0) 0x7f1344006590 con 0x7f13440090e0 2024-09-18T11:46:19.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1344009670 con 0x7f13440090e0 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1360005850 con 0x7f13440090e0 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 10 monclient: dump: 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.518 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 464068235 0 0) 0x7f1344009670 con 0x7f13440090e0 2024-09-18T11:46:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1344009930 con 0x7f13440090e0 2024-09-18T11:46:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 1289295110) 0x7f1360005ab0 con 0x7f13440090e0 2024-09-18T11:46:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:46:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6789/0 conn(0x7f13440090e0 legacy=0x7f1344008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f1355ffb700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6789/0 conn(0x7f13440090e0 legacy=0x7f1344008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 14, 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:46:19.520 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.521 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.522 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:46:19.523 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:46:19.524 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.525 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:46:19.526 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:46:19.527 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:19.528 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:19.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.530 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 7 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:19.531 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:19.532 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:19.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:19.532 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:19.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:19.532 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:46:19.532 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:19.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 10 librados: watch_flush enter 2024-09-18T11:46:19.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 10 librados: watch_flush exit 2024-09-18T11:46:19.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 20 client.29222.objecter shutdown clearing up homeless session... 2024-09-18T11:46:19.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 10 client.29222.objecter successfully canceled tick 2024-09-18T11:46:19.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.129:6812/20806 conn(0x7f1348038750 legacy=0x7f134803ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 10 monclient: shutdown 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 >> v1:172.21.15.112:6790/0 conn(0x7f1364132e40 legacy=0x7f136409e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136a08d700 1 -- 172.21.15.112:0/1465748618 reap_dead start 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136a08d700 1 -- 172.21.15.112:0/1465748618 reap_dead start 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 shutdown_connections 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 >> 172.21.15.112:0/1465748618 conn(0x7f13640a5b70 msgr2=0x7f136413e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 shutdown_connections 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 1 -- 172.21.15.112:0/1465748618 wait complete. 2024-09-18T11:46:19.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.515+0000 7f136baf0700 1 librados: shutdown 2024-09-18T11:46:19.538 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:46:19.538 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T11:46:19.538 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T11:46:19.538 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T11:46:19.538 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-09-18T11:46:19.538 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-09-18T11:46:19.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient: get_monmap_and_config 2024-09-18T11:46:19.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient: monmap: 2024-09-18T11:46:19.658 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.658 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.658 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:19.657406+0000 2024-09-18T11:46:19.659 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:19.657406+0000 2024-09-18T11:46:19.659 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.659 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.659 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.659 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:19.659 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:19.659 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 1 Processor -- start 2024-09-18T11:46:19.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 1 -- start start 2024-09-18T11:46:19.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient: init 2024-09-18T11:46:19.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:46:19.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient(hunting): picked mon.noname-c con 0x7f6e180a4300 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient(hunting): picked mon.noname-b con 0x7f6e1806e940 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient(hunting): picked mon.noname-a con 0x7f6e1806eda0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1806a8c0 con 0x7f6e1806eda0 2024-09-18T11:46:19.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e18132a40 con 0x7f6e180a4300 2024-09-18T11:46:19.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e180a46d0 con 0x7f6e1806e940 2024-09-18T11:46:19.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e1cd62700 10 monclient(hunting): authenticate will time out at 951.499341s 2024-09-18T11:46:19.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e16d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6e1806eda0 0x7f6e1813ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47158/0 (socket says 172.21.15.112:47158) 2024-09-18T11:46:19.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e16d9d700 1 -- 172.21.15.112:0/1617756844 learned_addr learned my addr 172.21.15.112:0/1617756844 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.655+0000 7f6e15d9b700 1 --1- 172.21.15.112:0/1617756844 >> v1:172.21.15.112:6790/0 conn(0x7f6e1806e940 0x7f6e18137660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34376/0 (socket says 172.21.15.112:34376) 2024-09-18T11:46:19.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e16d9d700 1 -- 172.21.15.112:0/1617756844 >> v1:172.21.15.112:6789/0 conn(0x7f6e1806eda0 legacy=0x7f6e1813ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e15d9b700 1 -- 172.21.15.112:0/1617756844 >> v1:172.21.15.112:6790/0 conn(0x7f6e1806e940 legacy=0x7f6e18137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1659c700 1 -- 172.21.15.112:0/1617756844 >> v1:172.21.15.129:6789/0 conn(0x7f6e180a4300 legacy=0x7f6e18132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291837052 0 0) 0x7f6e180a46d0 con 0x7f6e1806e940 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): my global_id is 29225 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e00003570 con 0x7f6e1806e940 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291382485 0 0) 0x7f6e1806a8c0 con 0x7f6e1806eda0 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): my global_id is 74166 2024-09-18T11:46:19.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e180a46d0 con 0x7f6e1806eda0 2024-09-18T11:46:19.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863044904 0 0) 0x7f6e18132a40 con 0x7f6e180a4300 2024-09-18T11:46:19.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient(hunting): my global_id is 17434 2024-09-18T11:46:19.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1806a8c0 con 0x7f6e180a4300 2024-09-18T11:46:19.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1097707225 0 0) 0x7f6e00003570 con 0x7f6e1806e940 2024-09-18T11:46:19.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 >> v1:172.21.15.129:6789/0 conn(0x7f6e180a4300 legacy=0x7f6e18132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 >> v1:172.21.15.112:6789/0 conn(0x7f6e1806eda0 legacy=0x7f6e1813ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 monclient: found mon.noname-b 2024-09-18T11:46:19.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1809fb50 con 0x7f6e1806e940 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e0c001870 con 0x7f6e1806e940 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 5 monclient: authenticate success, global_id 29225 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:46:19.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: dump: 2024-09-18T11:46:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.674 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e0c001b00 con 0x7f6e1806e940 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 1 -- 172.21.15.112:0/1617756844 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e0c004830 con 0x7f6e1806e940 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 4 set_mon_vals no callback set 2024-09-18T11:46:19.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1559a700 10 monclient: dump: 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 10 monclient: shutdown 2024-09-18T11:46:19.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1617756844 >> v1:172.21.15.112:6790/0 conn(0x7f6e1806e940 legacy=0x7f6e18137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1617756844 shutdown_connections 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1617756844 >> 172.21.15.112:0/1617756844 conn(0x7f6e18067e30 msgr2=0x7f6e180a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1617756844 shutdown_connections 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.659+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1617756844 wait complete. 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient: monmap: 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.677 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 librados: starting msgr at 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 librados: starting objecter 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 Processor -- start 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 -- start start 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 librados: setting wanted keys 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 librados: calling monclient init 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient: init 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient(hunting): picked mon.noname-a con 0x7f6e1806e940 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient(hunting): picked mon.noname-b con 0x7f6e1806eda0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient(hunting): picked mon.noname-c con 0x7f6e180a4300 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e180a2a20 con 0x7f6e1806e940 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e18138e20 con 0x7f6e1806eda0 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1813a0f0 con 0x7f6e180a4300 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e1cd62700 10 monclient(hunting): authenticate will time out at 951.506333s 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e16d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f6e180a4300 0x7f6e18149cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34380/0 (socket says 172.21.15.112:34380) 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.663+0000 7f6e16d9d700 1 -- 172.21.15.112:0/1118776541 learned_addr learned my addr 172.21.15.112:0/1118776541 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1659c700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6789/0 conn(0x7f6e1806e940 legacy=0x7f6e18138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 client.?.objecter ms_handle_connect 0x7f6e1806e940 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 client.?.objecter ms_handle_connect 0x7f6e180a4300 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e16d9d700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6790/0 conn(0x7f6e180a4300 legacy=0x7f6e18149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 client.?.objecter ms_handle_connect 0x7f6e1806eda0 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e15d9b700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.129:6789/0 conn(0x7f6e1806eda0 legacy=0x7f6e18138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720531483 0 0) 0x7f6e180a2a20 con 0x7f6e1806e940 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): my global_id is 74169 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df8004f10 con 0x7f6e1806e940 2024-09-18T11:46:19.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868728474 0 0) 0x7f6e1813a0f0 con 0x7f6e180a4300 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): my global_id is 29228 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e180a2a20 con 0x7f6e180a4300 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250664212 0 0) 0x7f6e18138e20 con 0x7f6e1806eda0 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): my global_id is 17437 2024-09-18T11:46:19.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1813a0f0 con 0x7f6e1806eda0 2024-09-18T11:46:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159323353 0 0) 0x7f6df8004f10 con 0x7f6e1806e940 2024-09-18T11:46:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e18138e20 con 0x7f6e1806e940 2024-09-18T11:46:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e08001cc0 con 0x7f6e1806e940 2024-09-18T11:46:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): dump: 2024-09-18T11:46:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.685 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811561508 0 0) 0x7f6e180a2a20 con 0x7f6e180a4300 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6df8004f10 con 0x7f6e180a4300 2024-09-18T11:46:19.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e10002030 con 0x7f6e180a4300 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient(hunting): dump: 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 61559145 0 0) 0x7f6e1813a0f0 con 0x7f6e1806eda0 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e180a2a20 con 0x7f6e1806eda0 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2299042035 0 0) 0x7f6e18138e20 con 0x7f6e1806e940 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6790/0 conn(0x7f6e180a4300 legacy=0x7f6e18149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.129:6789/0 conn(0x7f6e1806eda0 legacy=0x7f6e18138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 monclient: found mon.a 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e18069930 con 0x7f6e1806e940 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 5 monclient: authenticate success, global_id 74169 2024-09-18T11:46:19.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 10 monclient: _renew_subs 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1813b6a0 con 0x7f6e1806e940 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 10 client.74169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 10 monclient: _renew_subs 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e18132750 con 0x7f6e1806e940 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 1 librados: init done 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e080021b0 con 0x7f6e1806e940 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e080024c0 con 0x7f6e1806e940 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e14d99700 4 set_mon_vals no callback set 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 10 monclient: dump: 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f6e08002690 con 0x7f6e1806e940 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.667+0000 7f6e1cd62700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6dfc006590 con 0x7f6dfc002840 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7f6e080442f0 con 0x7f6e1806e940 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 client.74169.objecter ms_dispatch 0x7f6e18000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 3 client.74169.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 3 client.74169.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 20 client.74169.objecter dump_active .. 0 homeless 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e16d9d700 10 monclient: get_auth_request con 0x7f6df80385b0 auth_method 0 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 client.74169.objecter ms_handle_connect 0x7f6dfc002840 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 client.74169.objecter resend_mon_ops 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e15d9b700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6790/0 conn(0x7f6dfc002840 legacy=0x7f6dfc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881496321 0 0) 0x7f6dfc006590 con 0x7f6dfc002840 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 monclient(hunting): my global_id is 29231 2024-09-18T11:46:19.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e18138e20 con 0x7f6dfc002840 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2948641169 0 0) 0x7f6e18138e20 con 0x7f6dfc002840 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dfc006590 con 0x7f6dfc002840 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e0c004ca0 con 0x7f6dfc002840 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 monclient: dump: 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.692 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 10 client.74169.objecter ms_handle_connect 0x7f6df80385b0 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 741769298 0 0) 0x7f6dfc006590 con 0x7f6dfc002840 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.671+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dfc00a140 con 0x7f6dfc002840 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.675+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6e0c004ff0 con 0x7f6dfc002840 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.675+0000 7f6e067fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.675+0000 7f6e067fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.675+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6790/0 conn(0x7f6dfc002840 legacy=0x7f6dfc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.675+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6790/0 conn(0x7f6dfc002840 legacy=0x7f6dfc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.683+0000 7f6e1cd62700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.683+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6dfc007570 con 0x7f6dfc006920 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 client.74169.objecter ms_handle_connect 0x7f6dfc006920 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 client.74169.objecter resend_mon_ops 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e15d9b700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6790/0 conn(0x7f6dfc006920 legacy=0x7f6dfc006da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903857211 0 0) 0x7f6dfc007570 con 0x7f6dfc006920 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 monclient(hunting): my global_id is 29234 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc006590 con 0x7f6dfc006920 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3925178253 0 0) 0x7f6dfc006590 con 0x7f6dfc006920 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dfc007570 con 0x7f6dfc006920 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e0c006ea0 con 0x7f6dfc006920 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 monclient: dump: 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.696 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 788274575 0 0) 0x7f6dfc007570 con 0x7f6dfc006920 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7f6dfc008800 con 0x7f6dfc006920 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7f6e0c0058d0 con 0x7f6dfc006920 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:19.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6790/0 conn(0x7f6dfc006920 legacy=0x7f6dfc006da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e067fc700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6790/0 conn(0x7f6dfc006920 legacy=0x7f6dfc006da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:19.698 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:19.698 INFO:teuthology.orchestra.run.smithi112.stdout: "ret": 0, 2024-09-18T11:46:19.698 INFO:teuthology.orchestra.run.smithi112.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-18T11:46:19.698 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:19.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 10 librados: watch_flush enter 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 10 librados: watch_flush exit 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 20 client.74169.objecter shutdown clearing up homeless session... 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 10 client.74169.objecter successfully canceled tick 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.129:6812/20806 conn(0x7f6df80385b0 legacy=0x7f6df803aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 10 monclient: shutdown 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 >> v1:172.21.15.112:6789/0 conn(0x7f6e1806e940 legacy=0x7f6e18138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e16d9d700 1 -- 172.21.15.112:0/1118776541 reap_dead start 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e16d9d700 1 -- 172.21.15.112:0/1118776541 reap_dead start 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 shutdown_connections 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 >> 172.21.15.112:0/1118776541 conn(0x7f6e18067e30 msgr2=0x7f6e1809ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 shutdown_connections 2024-09-18T11:46:19.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 1 -- 172.21.15.112:0/1118776541 wait complete. 2024-09-18T11:46:19.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.687+0000 7f6e1cd62700 1 librados: shutdown 2024-09-18T11:46:19.705 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T11:46:19.705 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T11:46:19.705 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:46:19.720 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T11:46:19.720 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T11:46:19.721 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:46:19.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient: get_monmap_and_config 2024-09-18T11:46:19.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient: monmap: 2024-09-18T11:46:19.826 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.826 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.826 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:19.825327+0000 2024-09-18T11:46:19.826 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:19.825327+0000 2024-09-18T11:46:19.827 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.827 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.827 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.827 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:19.827 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:19.827 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 1 Processor -- start 2024-09-18T11:46:19.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 1 -- start start 2024-09-18T11:46:19.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient: init 2024-09-18T11:46:19.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:46:19.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient(hunting): picked mon.noname-c con 0x7f21400a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient(hunting): picked mon.noname-a con 0x7f214012edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient(hunting): picked mon.noname-b con 0x7f214012f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21400981a0 con 0x7f214012edc0 2024-09-18T11:46:19.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214006c120 con 0x7f21400a2040 2024-09-18T11:46:19.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214006eef0 con 0x7f214012f8a0 2024-09-18T11:46:19.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2147028700 10 monclient(hunting): authenticate will time out at 951.667121s 2024-09-18T11:46:19.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2146827700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f214012f8a0 legacy=0x7f2140137230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:19.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2145825700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f214012edc0 0x7f214012f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47192/0 (socket says 172.21.15.112:47192) 2024-09-18T11:46:19.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.823+0000 7f2145825700 1 -- 172.21.15.112:0/1578826644 learned_addr learned my addr 172.21.15.112:0/1578826644 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145825700 1 -- 172.21.15.112:0/1578826644 >> v1:172.21.15.112:6789/0 conn(0x7f214012edc0 legacy=0x7f214012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2146026700 1 -- 172.21.15.112:0/1578826644 >> v1:172.21.15.129:6789/0 conn(0x7f21400a2040 legacy=0x7f2140097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951648561 0 0) 0x7f21400981a0 con 0x7f214012edc0 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient(hunting): my global_id is 74172 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c003570 con 0x7f214012edc0 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228931020 0 0) 0x7f214006c120 con 0x7f21400a2040 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient(hunting): my global_id is 17440 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21400981a0 con 0x7f21400a2040 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3147773068 0 0) 0x7f212c003570 con 0x7f214012edc0 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 >> v1:172.21.15.112:6790/0 conn(0x7f214012f8a0 legacy=0x7f2140137230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 >> v1:172.21.15.129:6789/0 conn(0x7f21400a2040 legacy=0x7f2140097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 monclient: found mon.noname-a 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140096dd0 con 0x7f214012edc0 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2147028700 5 monclient: authenticate success, global_id 74172 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2130001a00 con 0x7f214012edc0 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2147028700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:19.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: dump: 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2147028700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:19.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2130001be0 con 0x7f214012edc0 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2147028700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2147028700 4 set_mon_vals no callback set 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 1 -- 172.21.15.112:0/1578826644 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2130004a90 con 0x7f214012edc0 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2145024700 10 monclient: dump: 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.841 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2147028700 10 monclient: shutdown 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2147028700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.827+0000 7f2147028700 1 -- 172.21.15.112:0/1578826644 >> v1:172.21.15.112:6789/0 conn(0x7f214012edc0 legacy=0x7f214012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 -- 172.21.15.112:0/1578826644 shutdown_connections 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 -- 172.21.15.112:0/1578826644 >> 172.21.15.112:0/1578826644 conn(0x7f2140067da0 msgr2=0x7f21400a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 -- 172.21.15.112:0/1578826644 shutdown_connections 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 -- 172.21.15.112:0/1578826644 wait complete. 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient: build_initial_monmap 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient: monmap: 2024-09-18T11:46:19.842 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 librados: starting msgr at 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 librados: starting objecter 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 Processor -- start 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 -- start start 2024-09-18T11:46:19.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 librados: setting wanted keys 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 librados: calling monclient init 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient: init 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient(hunting): picked mon.noname-c con 0x7f21400a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient(hunting): picked mon.noname-b con 0x7f214012edc0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2146026700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f21400a2040 legacy=0x7f214013e070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient(hunting): picked mon.noname-a con 0x7f214012f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214012cf10 con 0x7f214012f8a0 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214013ef50 con 0x7f214012edc0 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214006f110 con 0x7f21400a2040 2024-09-18T11:46:19.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:19.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2147028700 10 monclient(hunting): authenticate will time out at 951.675615s 2024-09-18T11:46:19.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2146827700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f214012f8a0 0x7f2140146420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47194/0 (socket says 172.21.15.112:47194) 2024-09-18T11:46:19.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.831+0000 7f2146827700 1 -- 172.21.15.112:0/1113496857 learned_addr learned my addr 172.21.15.112:0/1113496857 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:19.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2146827700 1 -- 172.21.15.112:0/1113496857 >> v1:172.21.15.112:6789/0 conn(0x7f214012f8a0 legacy=0x7f2140146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 client.?.objecter ms_handle_connect 0x7f214012f8a0 2024-09-18T11:46:19.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 client.?.objecter ms_handle_connect 0x7f214012edc0 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2145825700 1 -- 172.21.15.112:0/1113496857 >> v1:172.21.15.129:6789/0 conn(0x7f214012edc0 legacy=0x7f2140137250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096557595 0 0) 0x7f214012cf10 con 0x7f214012f8a0 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): my global_id is 74175 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2124004f10 con 0x7f214012f8a0 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436524725 0 0) 0x7f214013ef50 con 0x7f214012edc0 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:19.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): my global_id is 17443 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f214012cf10 con 0x7f214012edc0 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3062450452 0 0) 0x7f2124004f10 con 0x7f214012f8a0 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f214013ef50 con 0x7f214012f8a0 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2138004740 con 0x7f214012f8a0 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): dump: 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.850 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47353073 0 0) 0x7f214012cf10 con 0x7f214012edc0 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2124004f10 con 0x7f214012edc0 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2130004830 con 0x7f214012edc0 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient(hunting): dump: 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.851 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4158612201 0 0) 0x7f214013ef50 con 0x7f214012f8a0 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 >> v1:172.21.15.112:6790/0 conn(0x7f21400a2040 legacy=0x7f214013e070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 >> v1:172.21.15.129:6789/0 conn(0x7f214012edc0 legacy=0x7f2140137250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:19.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient: _finish_hunting 0 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 monclient: found mon.a 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21401383d0 con 0x7f214012f8a0 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient: _finish_auth 0 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 5 monclient: authenticate success, global_id 74175 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 10 monclient: _renew_subs 2024-09-18T11:46:19.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214012d560 con 0x7f214012f8a0 2024-09-18T11:46:19.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 10 client.74175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:19.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 10 monclient: _renew_subs 2024-09-18T11:46:19.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f214012c990 con 0x7f214012f8a0 2024-09-18T11:46:19.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2147028700 1 librados: init done 2024-09-18T11:46:19.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21380049d0 con 0x7f214012f8a0 2024-09-18T11:46:19.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2138004da0 con 0x7f214012f8a0 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f2144823700 4 set_mon_vals no callback set 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.835+0000 7f21367fc700 10 monclient: dump: 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f2147028700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f2147028700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f2147028700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f2138004f00 con 0x7f214012f8a0 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21280050b0 con 0x7f214012f8a0 2024-09-18T11:46:19.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7f2138044640 con 0x7f214012f8a0 2024-09-18T11:46:19.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f21367fc700 10 client.74175.objecter ms_dispatch 0x7f2140000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:19.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f21367fc700 3 client.74175.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:19.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f21367fc700 3 client.74175.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:19.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f21367fc700 20 client.74175.objecter dump_active .. 0 homeless 2024-09-18T11:46:19.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.839+0000 7f2146026700 10 monclient: get_auth_request con 0x7f2124038400 auth_method 0 2024-09-18T11:46:19.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.843+0000 7f21367fc700 10 client.74175.objecter ms_handle_connect 0x7f2124038400 2024-09-18T11:46:19.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.851+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f21380051b0 con 0x7f214012f8a0 2024-09-18T11:46:19.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.851+0000 7f21367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:19.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:19.851+0000 7f21367fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:20.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:46:20.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:20.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:20.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f21280058e0 con 0x7f214012f8a0 2024-09-18T11:46:20.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f21367fc700 1 -- 172.21.15.112:0/1113496857 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2699104595) 0x7f21380147b0 con 0x7f214012f8a0 2024-09-18T11:46:20.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f21367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:20.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f21367fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:20.082 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:46:20.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 10 librados: watch_flush enter 2024-09-18T11:46:20.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 10 librados: watch_flush exit 2024-09-18T11:46:20.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 20 client.74175.objecter shutdown clearing up homeless session... 2024-09-18T11:46:20.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 10 client.74175.objecter successfully canceled tick 2024-09-18T11:46:20.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 >> v1:172.21.15.129:6812/20806 conn(0x7f2124038400 legacy=0x7f212403a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 10 monclient: shutdown 2024-09-18T11:46:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 >> v1:172.21.15.112:6789/0 conn(0x7f214012f8a0 legacy=0x7f2140146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 shutdown_connections 2024-09-18T11:46:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 >> 172.21.15.112:0/1113496857 conn(0x7f2140067da0 msgr2=0x7f214009fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.079+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 shutdown_connections 2024-09-18T11:46:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.083+0000 7f2147028700 1 -- 172.21.15.112:0/1113496857 wait complete. 2024-09-18T11:46:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:20.083+0000 7f2147028700 1 librados: shutdown 2024-09-18T11:46:23.097 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient: get_monmap_and_config 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient: build_initial_monmap 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient: monmap: 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:23.210754+0000 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:23.210754+0000 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:23.211 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:23.212 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:23.212 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:23.212 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 1 Processor -- start 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 1 -- start start 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient: init 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient(hunting): picked mon.noname-b con 0x7fc2340a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient(hunting): picked mon.noname-c con 0x7fc23406c280 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient(hunting): picked mon.noname-a con 0x7fc23406c670 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:23.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23406e950 con 0x7fc23406c670 2024-09-18T11:46:23.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2340a2270 con 0x7fc23406c280 2024-09-18T11:46:23.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23406d130 con 0x7fc2340a1ea0 2024-09-18T11:46:23.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:23.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23affa700 10 monclient(hunting): authenticate will time out at 955.051660s 2024-09-18T11:46:23.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc239ff8700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fc2340a1ea0 legacy=0x7fc23406bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:23.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.207+0000 7fc23a7f9700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc23406c670 0x7fc2341388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47206/0 (socket says 172.21.15.112:47206) 2024-09-18T11:46:23.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc23a7f9700 1 -- 172.21.15.112:0/4234746651 learned_addr learned my addr 172.21.15.112:0/4234746651 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:23.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc2397f7700 1 -- 172.21.15.112:0/4234746651 >> v1:172.21.15.129:6789/0 conn(0x7fc23406c280 legacy=0x7fc234135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:23.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc23a7f9700 1 -- 172.21.15.112:0/4234746651 >> v1:172.21.15.112:6789/0 conn(0x7fc23406c670 legacy=0x7fc2341388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:23.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312548064 0 0) 0x7fc23406e950 con 0x7fc23406c670 2024-09-18T11:46:23.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:23.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient(hunting): my global_id is 74178 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc220003570 con 0x7fc23406c670 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089117821 0 0) 0x7fc220003570 con 0x7fc23406c670 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 >> v1:172.21.15.112:6790/0 conn(0x7fc2340a1ea0 legacy=0x7fc23406bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 >> v1:172.21.15.129:6789/0 conn(0x7fc23406c280 legacy=0x7fc234135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: _finish_hunting 0 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 monclient: found mon.noname-a 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23409e370 con 0x7fc23406c670 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: _finish_auth 0 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc224001940 con 0x7fc23406c670 2024-09-18T11:46:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc23affa700 5 monclient: authenticate success, global_id 74178 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: dump: 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:23.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc23affa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc224001c10 con 0x7fc23406c670 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 1 -- 172.21.15.112:0/4234746651 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc224004a90 con 0x7fc23406c670 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc23affa700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc23affa700 4 set_mon_vals no callback set 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.211+0000 7fc238ff6700 10 monclient: dump: 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:23.221 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient: shutdown 2024-09-18T11:46:23.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- 172.21.15.112:0/4234746651 >> v1:172.21.15.112:6789/0 conn(0x7fc23406c670 legacy=0x7fc2341388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- 172.21.15.112:0/4234746651 shutdown_connections 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- 172.21.15.112:0/4234746651 >> 172.21.15.112:0/4234746651 conn(0x7fc234067da0 msgr2=0x7fc2340a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- 172.21.15.112:0/4234746651 shutdown_connections 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- 172.21.15.112:0/4234746651 wait complete. 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient: build_initial_monmap 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient: monmap: 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:23.223 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 librados: starting msgr at 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 librados: starting objecter 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 Processor -- start 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- start start 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 librados: setting wanted keys 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 librados: calling monclient init 2024-09-18T11:46:23.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient: init 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient(hunting): picked mon.noname-c con 0x7fc23406c280 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient(hunting): picked mon.noname-b con 0x7fc23406c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient(hunting): picked mon.noname-a con 0x7fc2340a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234132e00 con 0x7fc2340a1ea0 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234097fd0 con 0x7fc23406c670 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc239ff8700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fc23406c280 legacy=0x7fc234141430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2340982b0 con 0x7fc23406c280 2024-09-18T11:46:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23affa700 10 monclient(hunting): authenticate will time out at 955.058819s 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23a7f9700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc2340a1ea0 0x7fc234149780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47218/0 (socket says 172.21.15.112:47218) 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.215+0000 7fc23a7f9700 1 -- 172.21.15.112:0/2148554532 learned_addr learned my addr 172.21.15.112:0/2148554532 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23a7f9700 1 -- 172.21.15.112:0/2148554532 >> v1:172.21.15.112:6789/0 conn(0x7fc2340a1ea0 legacy=0x7fc234149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 client.?.objecter ms_handle_connect 0x7fc2340a1ea0 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 client.?.objecter ms_handle_connect 0x7fc23406c670 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc2397f7700 1 -- 172.21.15.112:0/2148554532 >> v1:172.21.15.129:6789/0 conn(0x7fc23406c670 legacy=0x7fc2341322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609801496 0 0) 0x7fc234132e00 con 0x7fc2340a1ea0 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient(hunting): my global_id is 74181 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc218004f10 con 0x7fc2340a1ea0 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 118567556 0 0) 0x7fc218004f10 con 0x7fc2340a1ea0 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc234132e00 con 0x7fc2340a1ea0 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc224001720 con 0x7fc2340a1ea0 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient(hunting): dump: 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:23.231 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2763191951 0 0) 0x7fc234132e00 con 0x7fc2340a1ea0 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 >> v1:172.21.15.112:6790/0 conn(0x7fc23406c280 legacy=0x7fc234141430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 >> v1:172.21.15.129:6789/0 conn(0x7fc23406c670 legacy=0x7fc2341322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 monclient: found mon.a 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:23.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc234098590 con 0x7fc2340a1ea0 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient: _finish_auth 0 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 5 monclient: authenticate success, global_id 74181 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 10 monclient: _renew_subs 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc234142230 con 0x7fc2340a1ea0 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 10 client.74181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 10 monclient: _renew_subs 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc234132b50 con 0x7fc2340a1ea0 2024-09-18T11:46:23.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc23affa700 1 librados: init done 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc224004f40 con 0x7fc2340a1ea0 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc2240050b0 con 0x7fc2340a1ea0 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc22bfff700 4 set_mon_vals no callback set 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.219+0000 7fc229ffb700 10 monclient: dump: 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:23.234 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc23affa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fc22400f930 con 0x7fc2340a1ea0 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc23affa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc23affa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:23.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2140050b0 con 0x7fc2340a1ea0 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7fc2240443e0 con 0x7fc2340a1ea0 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc229ffb700 10 client.74181.objecter ms_dispatch 0x7fc234000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc229ffb700 3 client.74181.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc229ffb700 3 client.74181.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc229ffb700 20 client.74181.objecter dump_active .. 0 homeless 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc239ff8700 10 monclient: get_auth_request con 0x7fc2180381c0 auth_method 0 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.223+0000 7fc229ffb700 10 client.74181.objecter ms_handle_connect 0x7fc2180381c0 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.231+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fc224014bb0 con 0x7fc2340a1ea0 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.231+0000 7fc229ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:23.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.231+0000 7fc229ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:23.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:46:23.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:23.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:23.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc2140058e0 con 0x7fc2340a1ea0 2024-09-18T11:46:23.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc229ffb700 1 -- 172.21.15.112:0/2148554532 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1123006459) 0x7fc2240145b0 con 0x7fc2340a1ea0 2024-09-18T11:46:23.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc229ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:23.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc229ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:23.454 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:46:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 10 librados: watch_flush enter 2024-09-18T11:46:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 10 librados: watch_flush exit 2024-09-18T11:46:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 20 client.74181.objecter shutdown clearing up homeless session... 2024-09-18T11:46:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 10 client.74181.objecter successfully canceled tick 2024-09-18T11:46:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 >> v1:172.21.15.129:6812/20806 conn(0x7fc2180381c0 legacy=0x7fc21803a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 10 monclient: shutdown 2024-09-18T11:46:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:23.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 >> v1:172.21.15.112:6789/0 conn(0x7fc2340a1ea0 legacy=0x7fc234149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:23.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 shutdown_connections 2024-09-18T11:46:23.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 >> 172.21.15.112:0/2148554532 conn(0x7fc234067da0 msgr2=0x7fc2340a0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:23.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 shutdown_connections 2024-09-18T11:46:23.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 1 -- 172.21.15.112:0/2148554532 wait complete. 2024-09-18T11:46:23.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:23.451+0000 7fc23affa700 1 librados: shutdown 2024-09-18T11:46:26.467 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:46:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.571+0000 7feb20801700 10 monclient: get_monmap_and_config 2024-09-18T11:46:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.571+0000 7feb20801700 10 monclient: build_initial_monmap 2024-09-18T11:46:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.571+0000 7feb20801700 10 monclient: monmap: 2024-09-18T11:46:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:26.576001+0000 2024-09-18T11:46:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:26.576001+0000 2024-09-18T11:46:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:26.577 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 1 Processor -- start 2024-09-18T11:46:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 1 -- start start 2024-09-18T11:46:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 10 monclient: init 2024-09-18T11:46:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:46:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 10 monclient(hunting): picked mon.noname-b con 0x7feb1c0a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 10 monclient(hunting): picked mon.noname-a con 0x7feb1c06cb80 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 10 monclient(hunting): picked mon.noname-c con 0x7feb1c06cfd0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c06f150 con 0x7feb1c06cb80 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c096de0 con 0x7feb1c06cfd0 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c06ee70 con 0x7feb1c0a2040 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb20801700 10 monclient(hunting): authenticate will time out at 958.417270s 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb1b7fe700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feb1c12e860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb1affd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7feb1c06cb80 0x7feb1c09e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:47232/0 (socket says 172.21.15.112:47232) 2024-09-18T11:46:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb1affd700 1 -- 172.21.15.112:0/3887928676 learned_addr learned my addr 172.21.15.112:0/3887928676 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb1affd700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6789/0 conn(0x7feb1c06cb80 legacy=0x7feb1c09e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.575+0000 7feb1bfff700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.129:6789/0 conn(0x7feb1c06cfd0 legacy=0x7feb1c137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:26.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:26.779+0000 7feb1b7fe700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feb1c12e860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:27.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:27.183+0000 7feb1b7fe700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feb1c12e860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:27.987+0000 7feb1b7fe700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feb1c12e860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:29.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 10 monclient(hunting): tick 2024-09-18T11:46:29.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 1 monclient(hunting): continuing hunt 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feb1c12e860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.129:6789/0 conn(0x7feb1c06cfd0 legacy=0x7feb1c137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6789/0 conn(0x7feb1c06cb80 legacy=0x7feb1c09e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 10 monclient(hunting): picked mon.noname-b con 0x7feb1c0a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 10 monclient(hunting): picked mon.noname-c con 0x7feafc006fa0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 10 monclient(hunting): picked mon.noname-a con 0x7feb1c06cb80 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c096b00 con 0x7feb1c06cb80 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c06ee70 con 0x7feafc006fa0 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feafc010530 con 0x7feb1c0a2040 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb197fa700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:29.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.575+0000 7feb1b7fe700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:29.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.579+0000 7feb1bfff700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6789/0 conn(0x7feb1c06cb80 legacy=0x7feafc00cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:29.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.579+0000 7feb1affd700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.129:6789/0 conn(0x7feafc006fa0 legacy=0x7feafc009400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:29.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:29.779+0000 7feb1b7fe700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:30.183+0000 7feb1b7fe700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:30.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:30.987+0000 7feb1b7fe700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 10 monclient(hunting): tick 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 1 monclient(hunting): continuing hunt 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.129:6789/0 conn(0x7feafc006fa0 legacy=0x7feafc009400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6789/0 conn(0x7feb1c06cb80 legacy=0x7feafc00cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 10 monclient(hunting): picked mon.noname-a con 0x7feafc0108f0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:32.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 10 monclient(hunting): picked mon.noname-c con 0x7feafc006fa0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:32.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 10 monclient(hunting): picked mon.noname-b con 0x7feb1c0a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:32.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feafc011790 con 0x7feafc0108f0 2024-09-18T11:46:32.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feafc010530 con 0x7feafc006fa0 2024-09-18T11:46:32.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 1 -- 172.21.15.112:0/3887928676 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feafc00bcb0 con 0x7feb1c0a2040 2024-09-18T11:46:32.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb197fa700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:32.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.574+0000 7feb1bfff700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:32.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.578+0000 7feb1b7fe700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6789/0 conn(0x7feafc0108f0 legacy=0x7feafc010cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:32.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.578+0000 7feb1affd700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.129:6789/0 conn(0x7feafc006fa0 legacy=0x7feafc009400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:32.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:32.778+0000 7feb1bfff700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:33.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.182+0000 7feb1bfff700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:33.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856346310 0 0) 0x7feafc010530 con 0x7feafc006fa0 2024-09-18T11:46:33.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:33.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:33.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient(hunting): my global_id is 17446 2024-09-18T11:46:33.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb08003570 con 0x7feafc006fa0 2024-09-18T11:46:33.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1271197190 0 0) 0x7feb08003570 con 0x7feafc006fa0 2024-09-18T11:46:33.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6790/0 conn(0x7feb1c0a2040 legacy=0x7feafc005860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:46:33.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.112:6789/0 conn(0x7feafc0108f0 legacy=0x7feafc010cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:33.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 monclient: found mon.noname-c 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feafc00cea0 con 0x7feafc006fa0 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: _finish_auth 0 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7feb0c001880 con 0x7feafc006fa0 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb20801700 5 monclient: authenticate success, global_id 17446 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb20801700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: dump: 2024-09-18T11:46:33.964 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb20801700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb0c001a90 con 0x7feafc006fa0 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:33.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 1 -- 172.21.15.112:0/3887928676 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7feb0c001da0 con 0x7feafc006fa0 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb20801700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb20801700 4 set_mon_vals no callback set 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.958+0000 7feb1a7fc700 10 monclient: dump: 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:33.966 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:33.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 10 monclient: shutdown 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 -- 172.21.15.112:0/3887928676 >> v1:172.21.15.129:6789/0 conn(0x7feafc006fa0 legacy=0x7feafc009400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 -- 172.21.15.112:0/3887928676 shutdown_connections 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 -- 172.21.15.112:0/3887928676 >> 172.21.15.112:0/3887928676 conn(0x7feb1c067da0 msgr2=0x7feb1c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 -- 172.21.15.112:0/3887928676 shutdown_connections 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 -- 172.21.15.112:0/3887928676 wait complete. 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 10 monclient: build_initial_monmap 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 10 monclient: monmap: 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:33.968 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:33.969 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:33.969 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:33.969 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:33.969 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:33.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 librados: starting msgr at 2024-09-18T11:46:33.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 librados: starting objecter 2024-09-18T11:46:33.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 Processor -- start 2024-09-18T11:46:33.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 -- start start 2024-09-18T11:46:33.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 librados: setting wanted keys 2024-09-18T11:46:33.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 1 librados: calling monclient init 2024-09-18T11:46:33.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.962+0000 7feb20801700 10 monclient: init 2024-09-18T11:46:33.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:33.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 10 monclient(hunting): picked mon.noname-c con 0x7feafc006fa0 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 10 monclient(hunting): picked mon.noname-a con 0x7feafc0108f0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 10 monclient(hunting): picked mon.noname-b con 0x7feb1c0a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c11c3e0 con 0x7feafc0108f0 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c12a3d0 con 0x7feb1c0a2040 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1c11c100 con 0x7feafc006fa0 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb20801700 10 monclient(hunting): authenticate will time out at 965.808839s 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb1b7fe700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7feafc006fa0 legacy=0x7feb1c127f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:33.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb1affd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7feafc0108f0 0x7feb1c146c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39722/0 (socket says 172.21.15.112:39722) 2024-09-18T11:46:33.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb1affd700 1 -- 172.21.15.112:0/2941560905 learned_addr learned my addr 172.21.15.112:0/2941560905 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 10 client.?.objecter ms_handle_connect 0x7feafc0108f0 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb1affd700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.112:6789/0 conn(0x7feafc0108f0 legacy=0x7feb1c146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 10 client.?.objecter ms_handle_connect 0x7feb1c0a2040 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb1bfff700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6789/0 conn(0x7feb1c0a2040 legacy=0x7feb1c14a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002927673 0 0) 0x7feb1c12a3d0 con 0x7feb1c0a2040 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 10 monclient(hunting): my global_id is 17449 2024-09-18T11:46:33.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.966+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feafc010530 con 0x7feb1c0a2040 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2268046521 0 0) 0x7feafc010530 con 0x7feb1c0a2040 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb1c12a3d0 con 0x7feb1c0a2040 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7feb14001d40 con 0x7feb1c0a2040 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient(hunting): dump: 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:33.974 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:33.975 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:33.975 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:33.975 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:33.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 998712482 0 0) 0x7feb1c12a3d0 con 0x7feb1c0a2040 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.112:6790/0 conn(0x7feafc006fa0 legacy=0x7feb1c127f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.112:6789/0 conn(0x7feafc0108f0 legacy=0x7feb1c146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient: _finish_hunting 0 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 monclient: found mon.b 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1c11c6c0 con 0x7feb1c0a2040 2024-09-18T11:46:33.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient: _finish_auth 0 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb20801700 5 monclient: authenticate success, global_id 17449 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb20801700 10 monclient: _renew_subs 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb20801700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb20801700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1c128680 con 0x7feb1c0a2040 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb1bfff700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6789/0 conn(0x7feb1c0a2040 legacy=0x7feb1c14a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb1bfff700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6789/0 conn(0x7feb1c0a2040 legacy=0x7feb1c14a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb1bfff700 1 --1- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6789/0 conn(0x7feb1c0a2040 0x7feb1c14a340 :-1 s=OPENED pgs=8573 cs=1 l=1).handle_message read tag failed 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb1bfff700 1 --1- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6789/0 conn(0x7feb1c0a2040 0x7feb1c14a340 :-1 s=OPENED pgs=8573 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient: ms_handle_reset current mon v1:172.21.15.129:6789/0 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6789/0 conn(0x7feb1c0a2040 legacy=0x7feb1c14a340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb20801700 10 client.17449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:33.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient(hunting): picked mon.b con 0x7feafc008b50 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient(hunting): picked mon.c con 0x7feafc006fa0 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient(hunting): picked mon.a con 0x7feafc0108f0 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb1affd700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.112:6790/0 conn(0x7feafc006fa0 legacy=0x7feafc00f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb1c12a3d0 con 0x7feafc0108f0 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb1c11c100 con 0x7feafc008b50 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feafc010530 con 0x7feafc006fa0 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb137fe700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb20801700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.970+0000 7feb20801700 1 librados: init done 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb1bfff700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.112:6789/0 conn(0x7feafc0108f0 legacy=0x7feafc00f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 client.17449.objecter ms_handle_connect 0x7feafc0108f0 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 client.17449.objecter resend_mon_ops 2024-09-18T11:46:33.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb20801700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:33.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 client.17449.objecter ms_handle_connect 0x7feafc008b50 2024-09-18T11:46:33.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 client.17449.objecter resend_mon_ops 2024-09-18T11:46:33.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb1b7fe700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6789/0 conn(0x7feafc008b50 legacy=0x7feafc008f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:33.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb20801700 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:33.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719323843 0 0) 0x7feb1c11c100 con 0x7feafc008b50 2024-09-18T11:46:33.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:33.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T11:46:33.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7feafc010280 con 0x7feafc008b50 2024-09-18T11:46:33.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 344994260 0 0) 0x7feafc010280 con 0x7feafc008b50 2024-09-18T11:46:33.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.112:6790/0 conn(0x7feafc006fa0 legacy=0x7feafc00f040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.112:6789/0 conn(0x7feafc0108f0 legacy=0x7feafc00f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: _finish_hunting 0 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 monclient: found mon.b 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7feafc00ffd0 con 0x7feafc008b50 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1c09df70 con 0x7feafc008b50 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1c09ea20 con 0x7feafc008b50 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:33.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feafc010530 con 0x7feafc008b50 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: _finish_auth 0 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7feb04004490 con 0x7feafc008b50 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: dump: 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:33.983 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb040045f0 con 0x7feafc008b50 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb137fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.974+0000 7feb19ffb700 4 set_mon_vals no callback set 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.978+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7feb0400ef00 con 0x7feafc008b50 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.978+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 6 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7feb04043ad0 con 0x7feafc008b50 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.978+0000 7feb137fe700 10 client.17449.objecter ms_dispatch 0x7feb1c000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.978+0000 7feb137fe700 3 client.17449.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.978+0000 7feb137fe700 3 client.17449.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.978+0000 7feb137fe700 20 client.17449.objecter dump_active .. 0 homeless 2024-09-18T11:46:33.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.978+0000 7feb1affd700 10 monclient: get_auth_request con 0x7feb1c0a2040 auth_method 0 2024-09-18T11:46:33.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.978+0000 7feb137fe700 10 client.17449.objecter ms_handle_connect 0x7feb1c0a2040 2024-09-18T11:46:33.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.986+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7feb04013e20 con 0x7feafc008b50 2024-09-18T11:46:33.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.986+0000 7feb137fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:33.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.986+0000 7feb137fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:33.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.994+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7feb04013c40 con 0x7feafc008b50 2024-09-18T11:46:33.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:33.994+0000 7feb137fe700 10 monclient: handle_mon_command_ack 1 not found 2024-09-18T11:46:34.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:46:34.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:34.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:46:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 1 -- 172.21.15.112:0/2941560905 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feb1c05f2e0 con 0x7feafc008b50 2024-09-18T11:46:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb137fe700 1 -- 172.21.15.112:0/2941560905 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2851931655) 0x7feb04013e20 con 0x7feafc008b50 2024-09-18T11:46:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb137fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:46:34.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb137fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:34.222 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:46:34.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 10 librados: watch_flush enter 2024-09-18T11:46:34.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 10 librados: watch_flush exit 2024-09-18T11:46:34.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 20 client.17449.objecter shutdown clearing up homeless session... 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 10 client.17449.objecter successfully canceled tick 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6812/20806 conn(0x7feb1c0a2040 legacy=0x7feafc046d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 10 monclient: shutdown 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.218+0000 7feb20801700 1 -- 172.21.15.112:0/2941560905 >> v1:172.21.15.129:6789/0 conn(0x7feafc008b50 legacy=0x7feafc008f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.222+0000 7feb20801700 1 -- 172.21.15.112:0/2941560905 shutdown_connections 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.222+0000 7feb20801700 1 -- 172.21.15.112:0/2941560905 >> 172.21.15.112:0/2941560905 conn(0x7feb1c067da0 msgr2=0x7feb1c12c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.222+0000 7feb20801700 1 -- 172.21.15.112:0/2941560905 shutdown_connections 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.222+0000 7feb20801700 1 -- 172.21.15.112:0/2941560905 wait complete. 2024-09-18T11:46:34.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.222+0000 7feb20801700 1 librados: shutdown 2024-09-18T11:46:34.235 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T11:46:34.235 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T11:46:34.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:46:34.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.338+0000 7f67fd36b700 10 monclient: get_monmap_and_config 2024-09-18T11:46:34.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.338+0000 7f67fd36b700 10 monclient: build_initial_monmap 2024-09-18T11:46:34.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.338+0000 7f67fd36b700 10 monclient: monmap: 2024-09-18T11:46:34.343 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:34.343257+0000 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:34.343257+0000 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:34.344 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.338+0000 7f67fd36b700 1 Processor -- start 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.338+0000 7f67fd36b700 1 -- start start 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.338+0000 7f67fd36b700 10 monclient: init 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 10 monclient(hunting): picked mon.noname-b con 0x7f67f80a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 10 monclient(hunting): picked mon.noname-c con 0x7f67f809e370 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 10 monclient(hunting): picked mon.noname-a con 0x7f67f8134740 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f806f150 con 0x7f67f8134740 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f8096de0 con 0x7f67f809e370 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f806ee70 con 0x7f67f80a2040 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 10 monclient(hunting): authenticate will time out at 966.184648s 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f7fff700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f67f80a2040 legacy=0x7f67f812e860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:34.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fcb6a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f67f8134740 0x7f67f8136b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39740/0 (socket says 172.21.15.112:39740) 2024-09-18T11:46:34.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fcb6a700 1 -- 172.21.15.112:0/3330464300 learned_addr learned my addr 172.21.15.112:0/3330464300 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:34.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fcb6a700 1 -- 172.21.15.112:0/3330464300 >> v1:172.21.15.112:6789/0 conn(0x7f67f8134740 legacy=0x7f67f8136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67effff700 1 -- 172.21.15.112:0/3330464300 >> v1:172.21.15.129:6789/0 conn(0x7f67f809e370 legacy=0x7f67f809e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336127272 0 0) 0x7f67f806f150 con 0x7f67f8134740 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient(hunting): my global_id is 74184 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d8003570 con 0x7f67f8134740 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824586304 0 0) 0x7f67f8096de0 con 0x7f67f809e370 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient(hunting): my global_id is 17452 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f806f150 con 0x7f67f809e370 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2026588875 0 0) 0x7f67d8003570 con 0x7f67f8134740 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 >> v1:172.21.15.112:6790/0 conn(0x7f67f80a2040 legacy=0x7f67f812e860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 >> v1:172.21.15.129:6789/0 conn(0x7f67f809e370 legacy=0x7f67f809e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient: _finish_hunting 0 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 monclient: found mon.noname-a 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f8096b00 con 0x7f67f8134740 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient: _finish_auth 0 2024-09-18T11:46:34.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67f0001d80 con 0x7f67f8134740 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 5 monclient: authenticate success, global_id 74184 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67f77fe700 10 monclient: dump: 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.352 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.342+0000 7f67fd36b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f0001f60 con 0x7f67f8134740 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67f77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67f77fe700 1 -- 172.21.15.112:0/3330464300 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67f0004e10 con 0x7f67f8134740 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 4 set_mon_vals no callback set 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67f77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67f77fe700 10 monclient: dump: 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.353 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 10 monclient: shutdown 2024-09-18T11:46:34.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:34.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 -- 172.21.15.112:0/3330464300 >> v1:172.21.15.112:6789/0 conn(0x7f67f8134740 legacy=0x7f67f8136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 -- 172.21.15.112:0/3330464300 shutdown_connections 2024-09-18T11:46:34.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 -- 172.21.15.112:0/3330464300 >> 172.21.15.112:0/3330464300 conn(0x7f67f8067da0 msgr2=0x7f67f80a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:34.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 -- 172.21.15.112:0/3330464300 shutdown_connections 2024-09-18T11:46:34.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 -- 172.21.15.112:0/3330464300 wait complete. 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 10 monclient: build_initial_monmap 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 10 monclient: monmap: 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 librados: starting msgr at 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 librados: starting objecter 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 Processor -- start 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 -- start start 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 librados: setting wanted keys 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 1 librados: calling monclient init 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.346+0000 7f67fd36b700 10 monclient: init 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient(hunting): picked mon.noname-b con 0x7f67f809e370 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient(hunting): picked mon.noname-a con 0x7f67f80a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient(hunting): picked mon.noname-c con 0x7f67f8134740 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f806e560 con 0x7f67f80a2040 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f80a0ed0 con 0x7f67f809e370 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f812a250 con 0x7f67f8134740 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient(hunting): authenticate will time out at 966.192374s 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fcb6a700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f67f8134740 legacy=0x7f67f8149c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67effff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f67f80a2040 0x7f67f8129b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39746/0 (socket says 172.21.15.112:39746) 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67effff700 1 -- 172.21.15.112:0/2467238077 learned_addr learned my addr 172.21.15.112:0/2467238077 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67effff700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6789/0 conn(0x7f67f80a2040 legacy=0x7f67f8129b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 client.?.objecter ms_handle_connect 0x7f67f80a2040 2024-09-18T11:46:34.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048425036 0 0) 0x7f67f806e560 con 0x7f67f80a2040 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f7fff700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.129:6789/0 conn(0x7f67f809e370 legacy=0x7f67f8129430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): my global_id is 74187 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67dc004f10 con 0x7f67f80a2040 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 client.?.objecter ms_handle_connect 0x7f67f809e370 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398000633 0 0) 0x7f67f80a0ed0 con 0x7f67f809e370 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): my global_id is 17455 2024-09-18T11:46:34.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f806e560 con 0x7f67f809e370 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3873242383 0 0) 0x7f67dc004f10 con 0x7f67f80a2040 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67f80a0ed0 con 0x7f67f80a2040 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67e8001f00 con 0x7f67f80a2040 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): dump: 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 197263262 0 0) 0x7f67f806e560 con 0x7f67f809e370 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67dc004f10 con 0x7f67f809e370 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67e0004410 con 0x7f67f809e370 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient(hunting): dump: 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.367 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3686400934 0 0) 0x7f67f80a0ed0 con 0x7f67f80a2040 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6790/0 conn(0x7f67f8134740 legacy=0x7f67f8149c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:46:34.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.129:6789/0 conn(0x7f67f809e370 legacy=0x7f67f8129430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 monclient: found mon.a 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f812bc70 con 0x7f67f80a2040 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 10 monclient: _finish_auth 0 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67f4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 5 monclient: authenticate success, global_id 74187 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient: _renew_subs 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.350+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f81342b0 con 0x7f67f80a2040 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67fd36b700 10 client.74187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:34.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67fd36b700 10 monclient: _renew_subs 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67fd36b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f809df80 con 0x7f67f80a2040 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67fd36b700 1 librados: init done 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67e8002390 con 0x7f67f80a2040 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67e80026a0 con 0x7f67f80a2040 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 10 monclient: dump: 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.370 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f6ffd700 4 set_mon_vals no callback set 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67fd36b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f67d0006590 con 0x7f67d0002840 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f67e8002930 con 0x7f67f80a2040 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7f67e80443e0 con 0x7f67f80a2040 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 10 client.74187.objecter ms_dispatch 0x7f67f8000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 3 client.74187.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:34.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 3 client.74187.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 20 client.74187.objecter dump_active .. 0 homeless 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67fcb6a700 10 monclient: get_auth_request con 0x7f67dc038260 auth_method 0 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 10 client.74187.objecter ms_handle_connect 0x7f67d0002840 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f4ff9700 10 client.74187.objecter resend_mon_ops 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.354+0000 7f67f7fff700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6789/0 conn(0x7f67d0002840 legacy=0x7f67d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669118391 0 0) 0x7f67d0006590 con 0x7f67d0002840 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 monclient(hunting): my global_id is 74190 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f80a0ed0 con 0x7f67d0002840 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4268937762 0 0) 0x7f67f80a0ed0 con 0x7f67d0002840 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d0006590 con 0x7f67d0002840 2024-09-18T11:46:34.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67e0004a00 con 0x7f67d0002840 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 monclient: dump: 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.373 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 client.74187.objecter ms_handle_connect 0x7f67dc038260 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2955448567 0 0) 0x7f67d0006590 con 0x7f67d0002840 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67d000a140 con 0x7f67d0002840 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f67e0004e50 con 0x7f67d0002840 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6789/0 conn(0x7f67d0002840 legacy=0x7f67d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.358+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6789/0 conn(0x7f67d0002840 legacy=0x7f67d0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:34.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.370+0000 7f67fd36b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:46:34.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.370+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f67d0009f10 con 0x7f67d0006900 2024-09-18T11:46:34.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.370+0000 7f67f7fff700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6789/0 conn(0x7f67d0006900 legacy=0x7f67d0006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.370+0000 7f67f4ff9700 10 client.74187.objecter ms_handle_connect 0x7f67d0006900 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 client.74187.objecter resend_mon_ops 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643916927 0 0) 0x7f67d0009f10 con 0x7f67d0006900 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 monclient(hunting): my global_id is 74193 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d0006590 con 0x7f67d0006900 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3230017003 0 0) 0x7f67d0006590 con 0x7f67d0006900 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67d0009f10 con 0x7f67d0006900 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67e0005450 con 0x7f67d0006900 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:34.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 monclient: dump: 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3041199488 0 0) 0x7f67d0009f10 con 0x7f67d0006900 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f67d0008760 con 0x7f67d0006900 2024-09-18T11:46:34.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3992 (unknown 0 0 2101105355) 0x7f67e0006810 con 0x7f67d0006900 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6789/0 conn(0x7f67d0006900 legacy=0x7f67d0006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67f4ff9700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6789/0 conn(0x7f67d0006900 legacy=0x7f67d0006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 18, 2024-09-18T11:46:34.380 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: 1 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:34.382 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:34.383 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:34.384 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:34.385 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:34.386 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:34.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.389 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.390 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:46:34.391 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:34.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 10 librados: watch_flush enter 2024-09-18T11:46:34.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 10 librados: watch_flush exit 2024-09-18T11:46:34.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 20 client.74187.objecter shutdown clearing up homeless session... 2024-09-18T11:46:34.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 10 client.74187.objecter successfully canceled tick 2024-09-18T11:46:34.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.129:6812/20806 conn(0x7f67dc038260 legacy=0x7f67dc03a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 10 monclient: shutdown 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 >> v1:172.21.15.112:6789/0 conn(0x7f67f80a2040 legacy=0x7f67f8129b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fcb6a700 1 -- 172.21.15.112:0/2467238077 reap_dead start 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fcb6a700 1 -- 172.21.15.112:0/2467238077 reap_dead start 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 shutdown_connections 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 >> 172.21.15.112:0/2467238077 conn(0x7f67f8067da0 msgr2=0x7f67f806bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 shutdown_connections 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 1 -- 172.21.15.112:0/2467238077 wait complete. 2024-09-18T11:46:34.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.374+0000 7f67fd36b700 1 librados: shutdown 2024-09-18T11:46:34.394 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:46:34.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T11:46:34.395 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:46:34.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient: get_monmap_and_config 2024-09-18T11:46:34.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient: build_initial_monmap 2024-09-18T11:46:34.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient: monmap: 2024-09-18T11:46:34.501 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:34.501 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.502 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:46:34.501192+0000 2024-09-18T11:46:34.502 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:46:34.501192+0000 2024-09-18T11:46:34.502 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:34.502 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.502 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:34.502 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:46:34.502 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:46:34.502 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 1 Processor -- start 2024-09-18T11:46:34.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 1 -- start start 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient: init 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient(hunting): picked mon.noname-c con 0x7f2fb806add0 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient(hunting): picked mon.noname-b con 0x7f2fb8132c30 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient(hunting): picked mon.noname-a con 0x7f2fb8133070 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb80a17f0 con 0x7f2fb8133070 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb8097010 con 0x7f2fb806add0 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb80972f0 con 0x7f2fb8132c30 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbd449700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f2fb8132c30 legacy=0x7f2fb8137850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbfeae700 10 monclient(hunting): authenticate will time out at 966.342982s 2024-09-18T11:46:34.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbe44b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f2fb8133070 0x7f2fb813af80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39762/0 (socket says 172.21.15.112:39762) 2024-09-18T11:46:34.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.498+0000 7f2fbe44b700 1 -- 172.21.15.112:0/4240470081 learned_addr learned my addr 172.21.15.112:0/4240470081 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:34.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbe44b700 1 -- 172.21.15.112:0/4240470081 >> v1:172.21.15.112:6789/0 conn(0x7f2fb8133070 legacy=0x7f2fb813af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765725397 0 0) 0x7f2fb80a17f0 con 0x7f2fb8133070 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient(hunting): my global_id is 74196 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbdc4a700 1 -- 172.21.15.112:0/4240470081 >> v1:172.21.15.129:6789/0 conn(0x7f2fb806add0 legacy=0x7f2fb8132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa4003570 con 0x7f2fb8133070 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3329986912 0 0) 0x7f2fa4003570 con 0x7f2fb8133070 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 >> v1:172.21.15.112:6790/0 conn(0x7f2fb8132c30 legacy=0x7f2fb8137850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 >> v1:172.21.15.129:6789/0 conn(0x7f2fb806add0 legacy=0x7f2fb8132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: _finish_hunting 0 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 monclient: found mon.noname-a 2024-09-18T11:46:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80975d0 con 0x7f2fb8133070 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: _finish_auth 0 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2fb0001a00 con 0x7f2fb8133070 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 5 monclient: authenticate success, global_id 74196 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: dump: 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.510 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fb0001be0 con 0x7f2fb8133070 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 10 monclient: get_monmap_and_config success 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 4 set_mon_vals no callback set 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 1 -- 172.21.15.112:0/4240470081 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2fb0004a90 con 0x7f2fb8133070 2024-09-18T11:46:34.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbcc48700 10 monclient: dump: 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.512 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 10 monclient: shutdown 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 -- 172.21.15.112:0/4240470081 >> v1:172.21.15.112:6789/0 conn(0x7f2fb8133070 legacy=0x7f2fb813af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 -- 172.21.15.112:0/4240470081 shutdown_connections 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 -- 172.21.15.112:0/4240470081 >> 172.21.15.112:0/4240470081 conn(0x7f2fb80a5f90 msgr2=0x7f2fb80a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 -- 172.21.15.112:0/4240470081 shutdown_connections 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 -- 172.21.15.112:0/4240470081 wait complete. 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 10 monclient: build_initial_monmap 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 10 monclient: monmap: 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:46:34.513 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 librados: starting msgr at 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 librados: starting objecter 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 Processor -- start 2024-09-18T11:46:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 -- start start 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 librados: setting wanted keys 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 1 librados: calling monclient init 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.502+0000 7f2fbfeae700 10 monclient: init 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient: _reopen_session rank -1 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient(hunting): picked mon.noname-c con 0x7f2fb806add0 addr v1:172.21.15.112:6790/0 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient(hunting): picked mon.noname-a con 0x7f2fb8132c30 addr v1:172.21.15.112:6789/0 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient(hunting): picked mon.noname-b con 0x7f2fb8133070 addr v1:172.21.15.129:6789/0 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb81202d0 con 0x7f2fb8132c30 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb8132570 con 0x7f2fb8133070 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb80a2270 con 0x7f2fb806add0 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbdc4a700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f2fb806add0 legacy=0x7f2fb81314b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:46:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient(hunting): _renew_subs 2024-09-18T11:46:34.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient(hunting): authenticate will time out at 966.348266s 2024-09-18T11:46:34.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbd449700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f2fb8132c30 0x7f2fb8146850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39764/0 (socket says 172.21.15.112:39764) 2024-09-18T11:46:34.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbd449700 1 -- 172.21.15.112:0/3222825582 learned_addr learned my addr 172.21.15.112:0/3222825582 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:46:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbd449700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.112:6789/0 conn(0x7f2fb8132c30 legacy=0x7f2fb8146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 client.?.objecter ms_handle_connect 0x7f2fb8132c30 2024-09-18T11:46:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748833495 0 0) 0x7f2fb81202d0 con 0x7f2fb8132c30 2024-09-18T11:46:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): my global_id is 74199 2024-09-18T11:46:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c004f10 con 0x7f2fb8132c30 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 client.?.objecter ms_handle_connect 0x7f2fb8133070 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbe44b700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6789/0 conn(0x7f2fb8133070 legacy=0x7f2fb8149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934137724 0 0) 0x7f2fb8132570 con 0x7f2fb8133070 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): my global_id is 17461 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb81202d0 con 0x7f2fb8133070 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1926147109 0 0) 0x7f2f9c004f10 con 0x7f2fb8132c30 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb8132570 con 0x7f2fb8132c30 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2fa80045a0 con 0x7f2fb8132c30 2024-09-18T11:46:34.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): dump: 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:46:34.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1153911109 0 0) 0x7f2fb8132570 con 0x7f2fb8132c30 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.112:6790/0 conn(0x7f2fb806add0 legacy=0x7f2fb81314b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6789/0 conn(0x7f2fb8133070 legacy=0x7f2fb8149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient: _finish_hunting 0 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 monclient: found mon.a 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb8142490 con 0x7f2fb8132c30 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient: _finish_auth 0 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 5 monclient: authenticate success, global_id 74199 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient: _renew_subs 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:34.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb8121600 con 0x7f2fb8132c30 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 client.74199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient: _renew_subs 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb8131bc0 con 0x7f2fb8132c30 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fbfeae700 1 librados: init done 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fa8004780 con 0x7f2fb8132c30 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2fa8004ad0 con 0x7f2fb8132c30 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2faffff700 4 set_mon_vals no callback set 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.506+0000 7f2fadffb700 10 monclient: dump: 2024-09-18T11:46:34.524 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fbfeae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2f98006590 con 0x7f2f98002840 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f2fa8004d60 con 0x7f2fb8132c30 2024-09-18T11:46:34.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6516+0+0 (unknown 2617144419 0 0) 0x7f2fa80443a0 con 0x7f2fb8132c30 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 10 client.74199.objecter ms_dispatch 0x7f2fb8000fd0 osd_map(73..73 src has 1..73) v4 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 3 client.74199.objecter handle_osd_map got epochs [73,73] > 0 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 3 client.74199.objecter handle_osd_map decoding full epoch 73 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 20 client.74199.objecter dump_active .. 0 homeless 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fbe44b700 10 monclient: get_auth_request con 0x7f2f9c0380c0 auth_method 0 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 10 client.74199.objecter ms_handle_connect 0x7f2f98002840 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 10 client.74199.objecter resend_mon_ops 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fbdc4a700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6789/0 conn(0x7f2f98002840 legacy=0x7f2f98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914497682 0 0) 0x7f2f98006590 con 0x7f2f98002840 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 10 monclient(hunting): my global_id is 17464 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.510+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb8132570 con 0x7f2f98002840 2024-09-18T11:46:34.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1912132263 0 0) 0x7f2fb8132570 con 0x7f2f98002840 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f98006590 con 0x7f2f98002840 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2fb4001dc0 con 0x7f2f98002840 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 10 monclient: dump: 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.527 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3905259812 0 0) 0x7f2f98006590 con 0x7f2f98002840 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f9800a140 con 0x7f2f98002840 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 10 client.74199.objecter ms_handle_connect 0x7f2f9c0380c0 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2fb4005b90 con 0x7f2f98002840 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6789/0 conn(0x7f2f98002840 legacy=0x7f2f98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.514+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6789/0 conn(0x7f2f98002840 legacy=0x7f2f98004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:34.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.526+0000 7f2fbfeae700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:46:34.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.526+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2f98007550 con 0x7f2f98006900 2024-09-18T11:46:34.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 client.74199.objecter ms_handle_connect 0x7f2f98006900 2024-09-18T11:46:34.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbdc4a700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6789/0 conn(0x7f2f98006900 legacy=0x7f2f98006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:46:34.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 client.74199.objecter resend_mon_ops 2024-09-18T11:46:34.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134636776 0 0) 0x7f2f98007550 con 0x7f2f98006900 2024-09-18T11:46:34.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 monclient(hunting): my global_id is 17467 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f98006590 con 0x7f2f98006900 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1839746959 0 0) 0x7f2f98006590 con 0x7f2f98006900 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f98007550 con 0x7f2f98006900 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2fb4002610 con 0x7f2f98006900 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 monclient: dump: 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:46:34.534 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.535 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:46:34.535 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:46:34.535 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:46:34.535 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:46:34.535 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:46:34.535 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:46:34.535 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3471545628 0 0) 0x7f2f98007550 con 0x7f2f98006900 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2f980087b0 con 0x7f2f98006900 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4160 (unknown 0 0 849657917) 0x7f2fb4002870 con 0x7f2f98006900 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6789/0 conn(0x7f2f98006900 legacy=0x7f2f98006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fadffb700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6789/0 conn(0x7f2f98006900 legacy=0x7f2f98006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:46:34.536 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 18, 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: 1 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:46:34.537 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:46:34.538 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:46:34.539 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:46:34.540 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.541 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.542 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:46:34.543 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:46:34.544 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:46:34.545 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:46:34.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 9 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:46:34.547 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:46:34.548 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 10 librados: watch_flush enter 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 10 librados: watch_flush exit 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 20 client.74199.objecter shutdown clearing up homeless session... 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 10 client.74199.objecter successfully canceled tick 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.129:6812/20806 conn(0x7f2f9c0380c0 legacy=0x7f2f9c03a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 10 monclient: shutdown 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 >> v1:172.21.15.112:6789/0 conn(0x7f2fb8132c30 legacy=0x7f2fb8146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbe44b700 1 -- 172.21.15.112:0/3222825582 reap_dead start 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbe44b700 1 -- 172.21.15.112:0/3222825582 reap_dead start 2024-09-18T11:46:34.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 shutdown_connections 2024-09-18T11:46:34.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 >> 172.21.15.112:0/3222825582 conn(0x7f2fb80a5f90 msgr2=0x7f2fb80a04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:46:34.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 shutdown_connections 2024-09-18T11:46:34.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 1 -- 172.21.15.112:0/3222825582 wait complete. 2024-09-18T11:46:34.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:46:34.530+0000 7f2fbfeae700 1 librados: shutdown 2024-09-18T11:46:34.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:46:34.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-18T11:46:40.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: seed 23945 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (119 ms) 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (24 ms) 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (29 ms) 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (16 ms) 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-18T11:46:40.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (9 ms) 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (4 ms) 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (12 ms) 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2024-09-18T11:46:40.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (20 ms) 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (3976 ms) 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (280 ms) 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: done waiting, doing copies 2024-09-18T11:46:40.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: done waiting 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60454 ms) 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (12 ms) 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (5 ms) 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (8734 ms) 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-18T11:46:40.037 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-18T11:46:40.038 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-18T11:46:40.038 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-18T11:46:40.038 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-18T11:46:55.047 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 22 tests from LibRadosMisc api_snapshots_pp: Running main() from gmock_main.cc 2024-09-18T11:46:55.047 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-09-18T11:46:55.047 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-18T11:46:55.047 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-18T11:46:55.047 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: seed 24037 2024-09-18T11:46:55.047 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-18T11:46:55.047 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2031 ms) 2024-09-18T11:46:55.047 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2079 ms) 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1895 ms) 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2010 ms) 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3893 ms) 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11908 ms total) 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4155 ms) 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-18T11:46:55.048 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4527 ms) 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5418 ms) 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3917 ms) 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2054 ms) 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi129-24037-7 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17860 ms) 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (37931 ms total) 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-09-18T11:46:55.049 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (14 ms) 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5102 ms) 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5116 ms total) 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2960 ms) 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2013 ms) 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2011 ms) 2024-09-18T11:46:55.050 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-18T11:46:55.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2004 ms) 2024-09-18T11:46:55.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8988 ms total) 2024-09-18T11:46:55.051 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-09-18T11:47:01.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_p api_aio: Running main() from gmock_main.cc 2024-09-18T11:47:01.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] Global test environment set-up. 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2497 ms) 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3076 ms) 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2917 ms) 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3026 ms) 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-18T11:47:01.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2880 ms) 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3086 ms) 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3084 ms) 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3138 ms) 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3487 ms) 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2237 ms) 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-18T11:47:01.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3247 ms) 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2816 ms) 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3019 ms) 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2902 ms) 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.Flush (2996 ms) 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3017 ms) 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2992 ms) 2024-09-18T11:47:01.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3654 ms) 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (18081 ms) 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3026 ms) 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3079 ms) 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2940 ms) 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2966 ms) 2024-09-18T11:47:01.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-18T11:47:01.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3019 ms) 2024-09-18T11:47:01.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-18T11:47:01.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3011 ms) 2024-09-18T11:47:01.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 25 tests from LibRadosAio (90194 ms total) 2024-09-18T11:47:01.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: 2024-09-18T11:47:01.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-18T11:47:01.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-18T11:47:01.033 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6985 ms) 2024-09-18T11:47:06.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplPP (73710 ms total) 2024-09-18T11:47:06.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-09-18T11:47:06.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-18T11:47:06.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-18T11:47:06.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (172 ms) 2024-09-18T11:47:06.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (172 ms total) 2024-09-18T11:47:06.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (54 ms) 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (9 ms) 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (63 ms total) 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (19 ms) 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (15 ms) 2024-09-18T11:47:06.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (34 ms total) 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (33 ms) 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (7 ms) 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (40 ms total) 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1058 ms) 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1058 ms total) 2024-09-18T11:47:06.274 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-09-18T11:47:06.275 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-18T11:47:06.275 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (102307 ms total) 2024-09-18T11:47:06.275 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-18T11:47:13.022 INFO:tasks.workunit.client.0.smithi129.stdout:p: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-18T11:47:13.022 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-18T11:47:13.022 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4047 ms) 2024-09-18T11:47:13.022 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-18T11:47:13.022 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3950 ms) 2024-09-18T11:47:13.023 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-18T11:47:13.023 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4010 ms) 2024-09-18T11:47:13.023 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12007 ms total) 2024-09-18T11:47:13.023 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-09-18T11:47:13.023 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-18T11:47:13.023 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (109039 ms total) 2024-09-18T11:47:13.023 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-18T11:47:19.027 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-18T11:47:19.027 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-18T11:47:19.027 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-18T11:47:19.027 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-18T11:47:19.027 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-18T11:47:19.027 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3457 ms) 2024-09-18T11:47:19.027 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-18T11:47:19.027 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (18088 ms) 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6660 ms) 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2295 ms) 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3156 ms) 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2842 ms) 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2940 ms) 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-18T11:47:19.028 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2968 ms) 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2997 ms) 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3007 ms) 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2999 ms) 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3668 ms) 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (18953 ms) 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-18T11:47:19.029 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2147 ms) 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3010 ms) 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2961 ms) 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3012 ms) 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3016 ms) 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3021 ms) 2024-09-18T11:47:19.030 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-18T11:47:19.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2960 ms) 2024-09-18T11:47:19.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-18T11:47:19.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3050 ms) 2024-09-18T11:47:19.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-18T11:47:19.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2993 ms) 2024-09-18T11:47:19.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-18T11:47:19.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2955 ms) 2024-09-18T11:47:19.031 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-18T11:47:19.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3053 ms) 2024-09-18T11:47:19.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-18T11:47:19.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2956 ms) 2024-09-18T11:47:19.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-18T11:47:19.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2988 ms) 2024-09-18T11:47:19.032 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-18T11:48:04.552 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T11:48:04.553 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T11:48:04.553 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T11:48:04.553 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T11:48:04.557 INFO:tasks.ceph.mon.c:Started 2024-09-18T11:48:04.557 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:48:04.557 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:48:04.654 INFO:tasks.ceph.mon.c.smithi112.stderr:2024-09-18T11:48:04.648+0000 7fe0bfb08a00 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 94 -> 0 (sync_force? mon rebuild?) 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient: get_monmap_and_config 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient: build_initial_monmap 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient: monmap: 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:04.658801+0000 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:04.658801+0000 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:04.661 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 1 Processor -- start 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 1 -- start start 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient: init 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient(hunting): picked mon.noname-b con 0x7f092806a830 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient(hunting): picked mon.noname-a con 0x7f092809bcc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient(hunting): picked mon.noname-c con 0x7f092806b310 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092806c490 con 0x7f092809bcc0 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092806c9d0 con 0x7f092806b310 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092809fd20 con 0x7f092806a830 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:04.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.652+0000 7f092d0f6700 10 monclient(hunting): authenticate will time out at 1056.499955s 2024-09-18T11:48:04.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0926d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f092806a830 0x7f092806ac00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:32892/0 (socket says 172.21.15.112:32892) 2024-09-18T11:48:04.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0926d9d700 1 -- 172.21.15.112:0/3654807678 learned_addr learned my addr 172.21.15.112:0/3654807678 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0926d9d700 1 -- 172.21.15.112:0/3654807678 >> v1:172.21.15.112:6790/0 conn(0x7f092806a830 legacy=0x7f092806ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f092759e700 1 -- 172.21.15.112:0/3654807678 >> v1:172.21.15.129:6789/0 conn(0x7f092806b310 legacy=0x7f0928137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000673707 0 0) 0x7f092806c9d0 con 0x7f092806b310 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient(hunting): my global_id is 17806 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0914003570 con 0x7f092806b310 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1997431679 0 0) 0x7f0914003570 con 0x7f092806b310 2024-09-18T11:48:04.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 >> v1:172.21.15.112:6790/0 conn(0x7f092806a830 legacy=0x7f092806ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:04.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 >> v1:172.21.15.112:6789/0 conn(0x7f092809bcc0 legacy=0x7f0928134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T11:48:04.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:48:04.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 monclient: found mon.noname-c 2024-09-18T11:48:04.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:04.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:48:04.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092809ff90 con 0x7f092806b310 2024-09-18T11:48:04.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: _finish_auth 0 2024-09-18T11:48:04.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f092d0f6700 5 monclient: authenticate success, global_id 17806 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f092d0f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0918001990 con 0x7f092806b310 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: dump: 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:04.668 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:04.669 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:04.669 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:04.669 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:04.669 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:04.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f092d0f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:04.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0918001c60 con 0x7f092806b310 2024-09-18T11:48:04.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:04.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f092d0f6700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 1 -- 172.21.15.112:0/3654807678 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f09180020a0 con 0x7f092806b310 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f092d0f6700 4 set_mon_vals no callback set 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.656+0000 7f0925d9b700 10 monclient: dump: 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:04.670 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient: shutdown 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- 172.21.15.112:0/3654807678 >> v1:172.21.15.129:6789/0 conn(0x7f092806b310 legacy=0x7f0928137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- 172.21.15.112:0/3654807678 shutdown_connections 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- 172.21.15.112:0/3654807678 >> 172.21.15.112:0/3654807678 conn(0x7f0928068110 msgr2=0x7f09280a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- 172.21.15.112:0/3654807678 shutdown_connections 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- 172.21.15.112:0/3654807678 wait complete. 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient: build_initial_monmap 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient: monmap: 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 librados: starting msgr at 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 librados: starting objecter 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 Processor -- start 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- start start 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 librados: setting wanted keys 2024-09-18T11:48:04.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 librados: calling monclient init 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient: init 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient(hunting): picked mon.noname-b con 0x7f092806a830 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient(hunting): picked mon.noname-c con 0x7f092806b310 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient(hunting): picked mon.noname-a con 0x7f092809bcc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092806efd0 con 0x7f092809bcc0 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0928139960 con 0x7f092806a830 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092809c670 con 0x7f092806b310 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092d0f6700 10 monclient(hunting): authenticate will time out at 1056.505652s 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f092806b310 0x7f0928142460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:32902/0 (socket says 172.21.15.112:32902) 2024-09-18T11:48:04.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.660+0000 7f092659c700 1 -- 172.21.15.112:0/3259014196 learned_addr learned my addr 172.21.15.112:0/3259014196 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:04.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092659c700 1 -- 172.21.15.112:0/3259014196 >> v1:172.21.15.112:6790/0 conn(0x7f092806b310 legacy=0x7f0928142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:04.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 client.?.objecter ms_handle_connect 0x7f092806b310 2024-09-18T11:48:04.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:04.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 client.?.objecter ms_handle_connect 0x7f092806a830 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f0926d9d700 1 -- 172.21.15.112:0/3259014196 >> v1:172.21.15.129:6789/0 conn(0x7f092806a830 legacy=0x7f0928139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95087532 0 0) 0x7f0928139960 con 0x7f092806a830 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): my global_id is 17809 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0908004f10 con 0x7f092806a830 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 client.?.objecter ms_handle_connect 0x7f092809bcc0 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092759e700 1 -- 172.21.15.112:0/3259014196 >> v1:172.21.15.112:6789/0 conn(0x7f092809bcc0 legacy=0x7f0928145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:04.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710240529 0 0) 0x7f092806efd0 con 0x7f092809bcc0 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): my global_id is 74514 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0928139960 con 0x7f092809bcc0 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882998843 0 0) 0x7f0928139960 con 0x7f092809bcc0 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f092806efd0 con 0x7f092809bcc0 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0918001f00 con 0x7f092809bcc0 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): dump: 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:04.679 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3228480884 0 0) 0x7f092806efd0 con 0x7f092809bcc0 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 >> v1:172.21.15.112:6790/0 conn(0x7f092806b310 legacy=0x7f0928142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 >> v1:172.21.15.129:6789/0 conn(0x7f092806a830 legacy=0x7f0928139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient: _finish_hunting 0 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 monclient: found mon.a 2024-09-18T11:48:04.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092813a770 con 0x7f092809bcc0 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient: _finish_auth 0 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 5 monclient: authenticate success, global_id 74514 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 10 monclient: _renew_subs 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092811e2f0 con 0x7f092809bcc0 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 10 client.74514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 10 monclient: _renew_subs 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f092811e680 con 0x7f092809bcc0 2024-09-18T11:48:04.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092d0f6700 1 librados: init done 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0918002120 con 0x7f092809bcc0 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0918002710 con 0x7f092809bcc0 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f092559a700 4 set_mon_vals no callback set 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.664+0000 7f091effd700 10 monclient: dump: 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:04.682 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0918002970 con 0x7f092809bcc0 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f092d0f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f092d0f6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f092d0f6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090c0050b0 con 0x7f092809bcc0 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(163..163 src has 1..163) v4 ==== 6119+0+0 (unknown 4251182424 0 0) 0x7f0918044190 con 0x7f092809bcc0 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f091effd700 10 client.74514.objecter ms_dispatch 0x7f0928000fd0 osd_map(163..163 src has 1..163) v4 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f091effd700 3 client.74514.objecter handle_osd_map got epochs [163,163] > 0 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f091effd700 3 client.74514.objecter handle_osd_map decoding full epoch 163 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f091effd700 20 client.74514.objecter dump_active .. 0 homeless 2024-09-18T11:48:04.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.668+0000 7f0926d9d700 10 monclient: get_auth_request con 0x7f0908038250 auth_method 0 2024-09-18T11:48:04.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.672+0000 7f091effd700 10 client.74514.objecter ms_handle_connect 0x7f0908038250 2024-09-18T11:48:04.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.676+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f0918002c20 con 0x7f092809bcc0 2024-09-18T11:48:04.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.676+0000 7f091effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:04.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.676+0000 7f091effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:04.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.904+0000 7f092d0f6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:48:04.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.904+0000 7f092d0f6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:04.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.904+0000 7f092d0f6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:04.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.904+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f090c0058e0 con 0x7f092809bcc0 2024-09-18T11:48:04.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f091effd700 1 -- 172.21.15.112:0/3259014196 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 780655605) 0x7f09180144f0 con 0x7f092809bcc0 2024-09-18T11:48:04.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f091effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:04.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f091effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:04.913 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 10 librados: watch_flush enter 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 10 librados: watch_flush exit 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 20 client.74514.objecter shutdown clearing up homeless session... 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 10 client.74514.objecter successfully canceled tick 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 >> v1:172.21.15.129:6812/20806 conn(0x7f0908038250 legacy=0x7f090803a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 10 monclient: shutdown 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 >> v1:172.21.15.112:6789/0 conn(0x7f092809bcc0 legacy=0x7f0928145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 shutdown_connections 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 >> 172.21.15.112:0/3259014196 conn(0x7f0928068110 msgr2=0x7f09280a0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 shutdown_connections 2024-09-18T11:48:04.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 1 -- 172.21.15.112:0/3259014196 wait complete. 2024-09-18T11:48:04.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:04.908+0000 7f092d0f6700 1 librados: shutdown 2024-09-18T11:48:04.927 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:48:04.927 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:48:04.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:48:05.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.020+0000 7f4456b3f700 10 monclient: get_monmap_and_config 2024-09-18T11:48:05.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.020+0000 7f4456b3f700 10 monclient: build_initial_monmap 2024-09-18T11:48:05.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.020+0000 7f4456b3f700 10 monclient: monmap: 2024-09-18T11:48:05.028 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:05.028 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.028 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:05.027939+0000 2024-09-18T11:48:05.028 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:05.027939+0000 2024-09-18T11:48:05.029 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:05.029 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.029 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:05.029 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:05.029 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:05.029 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 1 Processor -- start 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 1 -- start start 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 10 monclient: init 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 10 monclient(hunting): picked mon.noname-b con 0x7f44500a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 10 monclient(hunting): picked mon.noname-c con 0x7f44500a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 10 monclient(hunting): picked mon.noname-a con 0x7f4450097970 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4450097550 con 0x7f4450097970 2024-09-18T11:48:05.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4450096cb0 con 0x7f44500a0e60 2024-09-18T11:48:05.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44500a1440 con 0x7f44500a0380 2024-09-18T11:48:05.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:05.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4456b3f700 10 monclient(hunting): authenticate will time out at 1056.869428s 2024-09-18T11:48:05.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4455b3d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f44500a0380 0x7f44500a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:32904/0 (socket says 172.21.15.112:32904) 2024-09-18T11:48:05.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4455b3d700 1 -- 172.21.15.112:0/2046760862 learned_addr learned my addr 172.21.15.112:0/2046760862 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:05.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f445633e700 1 -- 172.21.15.112:0/2046760862 >> v1:172.21.15.112:6789/0 conn(0x7f4450097970 legacy=0x7f445013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4455b3d700 1 -- 172.21.15.112:0/2046760862 >> v1:172.21.15.112:6790/0 conn(0x7f44500a0380 legacy=0x7f44500a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f445533c700 1 -- 172.21.15.112:0/2046760862 >> v1:172.21.15.129:6789/0 conn(0x7f44500a0e60 legacy=0x7f4450137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.024+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229444420 0 0) 0x7f4450097550 con 0x7f4450097970 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient(hunting): my global_id is 74517 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f443c003570 con 0x7f4450097970 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182173510 0 0) 0x7f4450096cb0 con 0x7f44500a0e60 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient(hunting): my global_id is 17812 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4450097550 con 0x7f44500a0e60 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1459009025 0 0) 0x7f443c003570 con 0x7f4450097970 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 >> v1:172.21.15.112:6790/0 conn(0x7f44500a0380 legacy=0x7f44500a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 >> v1:172.21.15.129:6789/0 conn(0x7f44500a0e60 legacy=0x7f4450137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: _finish_hunting 0 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 monclient: found mon.noname-a 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500a1de0 con 0x7f4450097970 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: _finish_auth 0 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 5 monclient: authenticate success, global_id 74517 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4448001d10 con 0x7f4450097970 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: dump: 2024-09-18T11:48:05.038 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:05.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4448001fa0 con 0x7f4450097970 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 1 -- 172.21.15.112:0/2046760862 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4448002420 con 0x7f4450097970 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 4 set_mon_vals no callback set 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4454b3b700 10 monclient: dump: 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.040 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 10 monclient: shutdown 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 -- 172.21.15.112:0/2046760862 >> v1:172.21.15.112:6789/0 conn(0x7f4450097970 legacy=0x7f445013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 -- 172.21.15.112:0/2046760862 shutdown_connections 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 -- 172.21.15.112:0/2046760862 >> 172.21.15.112:0/2046760862 conn(0x7f4450068100 msgr2=0x7f44500a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:05.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 -- 172.21.15.112:0/2046760862 shutdown_connections 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 -- 172.21.15.112:0/2046760862 wait complete. 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 10 monclient: build_initial_monmap 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 10 monclient: monmap: 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 librados: starting msgr at 2024-09-18T11:48:05.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 librados: starting objecter 2024-09-18T11:48:05.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 Processor -- start 2024-09-18T11:48:05.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 -- start start 2024-09-18T11:48:05.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 librados: setting wanted keys 2024-09-18T11:48:05.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 1 librados: calling monclient init 2024-09-18T11:48:05.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.028+0000 7f4456b3f700 10 monclient: init 2024-09-18T11:48:05.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:05.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 10 monclient(hunting): picked mon.noname-b con 0x7f4450097970 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 10 monclient(hunting): picked mon.noname-a con 0x7f44500a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 10 monclient(hunting): picked mon.noname-c con 0x7f44500a0e60 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4450135180 con 0x7f44500a0380 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44501393d0 con 0x7f4450097970 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44500a2050 con 0x7f44500a0e60 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4456b3f700 10 monclient(hunting): authenticate will time out at 1056.876909s 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f445533c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f44500a0380 0x7f44501467e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34012/0 (socket says 172.21.15.112:34012) 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f445533c700 1 -- 172.21.15.112:0/2215431575 learned_addr learned my addr 172.21.15.112:0/2215431575 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 client.?.objecter ms_handle_connect 0x7f44500a0380 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f445533c700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6789/0 conn(0x7f44500a0380 legacy=0x7f44501467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 client.?.objecter ms_handle_connect 0x7f44500a0e60 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f445633e700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6790/0 conn(0x7f44500a0e60 legacy=0x7f4450149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4455b3d700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.129:6789/0 conn(0x7f4450097970 legacy=0x7f4450141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 client.?.objecter ms_handle_connect 0x7f4450097970 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581531280 0 0) 0x7f4450135180 con 0x7f44500a0380 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): my global_id is 74520 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4434004f10 con 0x7f44500a0380 2024-09-18T11:48:05.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890184341 0 0) 0x7f44500a2050 con 0x7f44500a0e60 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): my global_id is 84101 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4450135180 con 0x7f44500a0e60 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056576766 0 0) 0x7f44501393d0 con 0x7f4450097970 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 10 monclient(hunting): my global_id is 17815 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44500a2050 con 0x7f4450097970 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.032+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918586259 0 0) 0x7f4434004f10 con 0x7f44500a0380 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44501393d0 con 0x7f44500a0380 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f444c001d20 con 0x7f44500a0380 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:05.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): dump: 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 635805196 0 0) 0x7f4450135180 con 0x7f44500a0e60 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4434004f10 con 0x7f44500a0e60 2024-09-18T11:48:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4448002420 con 0x7f44500a0e60 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): dump: 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3380816432 0 0) 0x7f44500a2050 con 0x7f4450097970 2024-09-18T11:48:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4450135180 con 0x7f4450097970 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4444001a60 con 0x7f4450097970 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient(hunting): dump: 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.054 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1256705182 0 0) 0x7f44501393d0 con 0x7f44500a0380 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6790/0 conn(0x7f44500a0e60 legacy=0x7f4450149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.129:6789/0 conn(0x7f4450097970 legacy=0x7f4450141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 monclient: found mon.a 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450142330 con 0x7f44500a0380 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient: _finish_auth 0 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 5 monclient: authenticate success, global_id 74520 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 10 monclient: _renew_subs 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445013aa20 con 0x7f44500a0380 2024-09-18T11:48:05.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 10 client.74520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 10 monclient: _renew_subs 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4450139a60 con 0x7f44500a0380 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 1 librados: init done 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f444c0022d0 con 0x7f44500a0380 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f444c002620 con 0x7f44500a0380 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4443fff700 4 set_mon_vals no callback set 2024-09-18T11:48:05.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 monclient: dump: 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.057 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4430006590 con 0x7f4430002840 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f444c002830 con 0x7f44500a0380 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(163..163 src has 1..163) v4 ==== 6119+0+0 (unknown 4251182424 0 0) 0x7f444c044130 con 0x7f44500a0380 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 10 client.74520.objecter ms_dispatch 0x7f4450000fd0 osd_map(163..163 src has 1..163) v4 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 3 client.74520.objecter handle_osd_map got epochs [163,163] > 0 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 3 client.74520.objecter handle_osd_map decoding full epoch 163 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f4441ffb700 20 client.74520.objecter dump_active .. 0 homeless 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.036+0000 7f445633e700 10 monclient: get_auth_request con 0x7f44340385d0 auth_method 0 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4455b3d700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6789/0 conn(0x7f4430002840 legacy=0x7f4430004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 client.74520.objecter ms_handle_connect 0x7f4430002840 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 client.74520.objecter resend_mon_ops 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276104346 0 0) 0x7f4430006590 con 0x7f4430002840 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 monclient(hunting): my global_id is 74523 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44501393d0 con 0x7f4430002840 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 202927766 0 0) 0x7f44501393d0 con 0x7f4430002840 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4430006590 con 0x7f4430002840 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f44440049d0 con 0x7f4430002840 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 monclient: dump: 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.059 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2890249147 0 0) 0x7f4430006590 con 0x7f4430002840 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f443000a140 con 0x7f4430002840 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 client.74520.objecter ms_handle_connect 0x7f44340385d0 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f4444004ce0 con 0x7f4430002840 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6789/0 conn(0x7f4430002840 legacy=0x7f4430004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.040+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6789/0 conn(0x7f4430002840 legacy=0x7f4430004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:05.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.048+0000 7f4456b3f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:48:05.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.048+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4430009670 con 0x7f44300090e0 2024-09-18T11:48:05.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 client.74520.objecter ms_handle_connect 0x7f44300090e0 2024-09-18T11:48:05.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 client.74520.objecter resend_mon_ops 2024-09-18T11:48:05.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4455b3d700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6789/0 conn(0x7f44300090e0 legacy=0x7f4430008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434883393 0 0) 0x7f4430009670 con 0x7f44300090e0 2024-09-18T11:48:05.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 monclient(hunting): my global_id is 74526 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4430006590 con 0x7f44300090e0 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3505751103 0 0) 0x7f4430006590 con 0x7f44300090e0 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4430009670 con 0x7f44300090e0 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f44440052e0 con 0x7f44300090e0 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 monclient: dump: 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.063 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4176196224 0 0) 0x7f4430009670 con 0x7f44300090e0 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4430009930 con 0x7f44300090e0 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4003 (unknown 0 0 2306348187) 0x7f4444006660 con 0x7f44300090e0 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6789/0 conn(0x7f44300090e0 legacy=0x7f4430008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4441ffb700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6789/0 conn(0x7f44300090e0 legacy=0x7f4430008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 20, 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:48:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:48:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:48:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:48:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.077 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:48:05.077 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:48:05.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 10 librados: watch_flush enter 2024-09-18T11:48:05.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 10 librados: watch_flush exit 2024-09-18T11:48:05.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 20 client.74520.objecter shutdown clearing up homeless session... 2024-09-18T11:48:05.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 10 client.74520.objecter successfully canceled tick 2024-09-18T11:48:05.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.129:6812/20806 conn(0x7f44340385d0 legacy=0x7f443403aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 10 monclient: shutdown 2024-09-18T11:48:05.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:05.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 >> v1:172.21.15.112:6789/0 conn(0x7f44500a0380 legacy=0x7f44501467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f445633e700 1 -- 172.21.15.112:0/2215431575 reap_dead start 2024-09-18T11:48:05.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f445633e700 1 -- 172.21.15.112:0/2215431575 reap_dead start 2024-09-18T11:48:05.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 shutdown_connections 2024-09-18T11:48:05.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 >> 172.21.15.112:0/2215431575 conn(0x7f4450068100 msgr2=0x7f44500a4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:05.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 shutdown_connections 2024-09-18T11:48:05.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 1 -- 172.21.15.112:0/2215431575 wait complete. 2024-09-18T11:48:05.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.052+0000 7f4456b3f700 1 librados: shutdown 2024-09-18T11:48:05.080 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:48:05.080 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:48:05.081 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient: get_monmap_and_config 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient: build_initial_monmap 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient: monmap: 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:05.189783+0000 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:05.189783+0000 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:05.191 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 1 Processor -- start 2024-09-18T11:48:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 1 -- start start 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient: init 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient(hunting): picked mon.noname-a con 0x7f40e80a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient(hunting): picked mon.noname-b con 0x7f40e80a0e60 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient(hunting): picked mon.noname-c con 0x7f40e8097970 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e8097550 con 0x7f40e80a0380 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e8096cb0 con 0x7f40e8097970 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e80a1440 con 0x7f40e80a0e60 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ee3ac700 10 monclient(hunting): authenticate will time out at 1057.031382s 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ed3aa700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f40e80a0380 0x7f40e80a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34040/0 (socket says 172.21.15.112:34040) 2024-09-18T11:48:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ecba9700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f40e80a0e60 0x7f40e8137390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:32930/0 (socket says 172.21.15.112:32930) 2024-09-18T11:48:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.184+0000 7f40ed3aa700 1 -- 172.21.15.112:0/1818567502 learned_addr learned my addr 172.21.15.112:0/1818567502 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ecba9700 1 -- 172.21.15.112:0/1818567502 >> v1:172.21.15.112:6790/0 conn(0x7f40e80a0e60 legacy=0x7f40e8137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ed3aa700 1 -- 172.21.15.112:0/1818567502 >> v1:172.21.15.112:6789/0 conn(0x7f40e80a0380 legacy=0x7f40e80a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925648422 0 0) 0x7f40e80a1440 con 0x7f40e80a0e60 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient(hunting): my global_id is 84104 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d4003570 con 0x7f40e80a0e60 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229701604 0 0) 0x7f40e8097550 con 0x7f40e80a0380 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient(hunting): my global_id is 74529 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40edbab700 1 -- 172.21.15.112:0/1818567502 >> v1:172.21.15.129:6789/0 conn(0x7f40e8097970 legacy=0x7f40e813aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e80a1440 con 0x7f40e80a0380 2024-09-18T11:48:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 749592869 0 0) 0x7f40d4003570 con 0x7f40e80a0e60 2024-09-18T11:48:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 >> v1:172.21.15.129:6789/0 conn(0x7f40e8097970 legacy=0x7f40e813aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 >> v1:172.21.15.112:6789/0 conn(0x7f40e80a0380 legacy=0x7f40e80a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: _finish_hunting 0 2024-09-18T11:48:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 monclient: found mon.noname-b 2024-09-18T11:48:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:48:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e80a1de0 con 0x7f40e80a0e60 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: _finish_auth 0 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 5 monclient: authenticate success, global_id 84104 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f40dc001d80 con 0x7f40e80a0e60 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: dump: 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40dc001f60 con 0x7f40e80a0e60 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 4 set_mon_vals no callback set 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 1 -- 172.21.15.112:0/1818567502 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f40dc004d50 con 0x7f40e80a0e60 2024-09-18T11:48:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40dbfff700 10 monclient: dump: 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 10 monclient: shutdown 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1818567502 >> v1:172.21.15.112:6790/0 conn(0x7f40e80a0e60 legacy=0x7f40e8137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.188+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1818567502 shutdown_connections 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1818567502 >> 172.21.15.112:0/1818567502 conn(0x7f40e8068100 msgr2=0x7f40e80a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1818567502 shutdown_connections 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1818567502 wait complete. 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient: build_initial_monmap 2024-09-18T11:48:05.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient: monmap: 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 librados: starting msgr at 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 librados: starting objecter 2024-09-18T11:48:05.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 Processor -- start 2024-09-18T11:48:05.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 -- start start 2024-09-18T11:48:05.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 librados: setting wanted keys 2024-09-18T11:48:05.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 librados: calling monclient init 2024-09-18T11:48:05.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient: init 2024-09-18T11:48:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:48:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient(hunting): picked mon.noname-a con 0x7f40e8097970 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient(hunting): picked mon.noname-c con 0x7f40e80a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient(hunting): picked mon.noname-b con 0x7f40e80a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e81226e0 con 0x7f40e8097970 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e813da70 con 0x7f40e80a0e60 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e8122400 con 0x7f40e80a0380 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ee3ac700 10 monclient(hunting): authenticate will time out at 1057.038356s 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ed3aa700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f40e8097970 0x7f40e813d360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34044/0 (socket says 172.21.15.112:34044) 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.192+0000 7f40ed3aa700 1 -- 172.21.15.112:0/1559230994 learned_addr learned my addr 172.21.15.112:0/1559230994 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ed3aa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6789/0 conn(0x7f40e8097970 legacy=0x7f40e813d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 client.?.objecter ms_handle_connect 0x7f40e8097970 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ecba9700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6790/0 conn(0x7f40e80a0380 legacy=0x7f40e81465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 client.?.objecter ms_handle_connect 0x7f40e80a0380 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 client.?.objecter ms_handle_connect 0x7f40e80a0e60 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ed3aa700 1 --1- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6789/0 conn(0x7f40e8097970 0x7f40e813d360 :-1 s=OPENED pgs=334 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f40e81226e0 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40edbab700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.129:6789/0 conn(0x7f40e80a0e60 legacy=0x7f40e8149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362202366 0 0) 0x7f40e8122400 con 0x7f40e80a0380 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): my global_id is 84107 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40cc004f10 con 0x7f40e80a0380 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754092610 0 0) 0x7f40e813da70 con 0x7f40e80a0e60 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): my global_id is 17821 2024-09-18T11:48:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e8122400 con 0x7f40e80a0e60 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684150400 0 0) 0x7f40cc004f10 con 0x7f40e80a0380 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40e813da70 con 0x7f40e80a0380 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f40dc001b20 con 0x7f40e80a0380 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): dump: 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 234496810 0 0) 0x7f40e813da70 con 0x7f40e80a0380 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.129:6789/0 conn(0x7f40e80a0e60 legacy=0x7f40e8149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6789/0 conn(0x7f40e8097970 legacy=0x7f40e813d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient: _finish_hunting 0 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 monclient: found mon.c 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e81229c0 con 0x7f40e80a0380 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient: _finish_auth 0 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 5 monclient: authenticate success, global_id 84107 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 10 monclient: _renew_subs 2024-09-18T11:48:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131227829 0 0) 0x7f40e81226e0 con 0x7f40e8097970 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e8120170 con 0x7f40e80a0380 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 10 client.84107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 10 monclient: _renew_subs 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e813e860 con 0x7f40e80a0380 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 1 librados: init done 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40dc001da0 con 0x7f40e80a0380 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f40dc005550 con 0x7f40e80a0380 2024-09-18T11:48:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40db7fe700 4 set_mon_vals no callback set 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40d97fa700 10 monclient: dump: 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.196+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f40c8006590 con 0x7f40c8002840 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f40dc00fdd0 con 0x7f40e80a0380 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(163..163 src has 1..163) v4 ==== 6119+0+0 (unknown 4251182424 0 0) 0x7f40dc044660 con 0x7f40e80a0380 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 client.84107.objecter ms_dispatch 0x7f40e8000fd0 osd_map(163..163 src has 1..163) v4 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 3 client.84107.objecter handle_osd_map got epochs [163,163] > 0 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 3 client.84107.objecter handle_osd_map decoding full epoch 163 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40edbab700 10 monclient: get_auth_request con 0x7f40cc038360 auth_method 0 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 20 client.84107.objecter dump_active .. 0 homeless 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40ed3aa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6790/0 conn(0x7f40c8002840 legacy=0x7f40c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 client.84107.objecter ms_handle_connect 0x7f40c8002840 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 client.84107.objecter resend_mon_ops 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230562144 0 0) 0x7f40c8006590 con 0x7f40c8002840 2024-09-18T11:48:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 monclient(hunting): my global_id is 84110 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e81226e0 con 0x7f40c8002840 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 271286439 0 0) 0x7f40e81226e0 con 0x7f40c8002840 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40c8006590 con 0x7f40c8002840 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f40e4001ba0 con 0x7f40c8002840 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 monclient: dump: 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 530675990 0 0) 0x7f40c8006590 con 0x7f40c8002840 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40c800a140 con 0x7f40c8002840 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f40e4001a00 con 0x7f40c8002840 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6790/0 conn(0x7f40c8002840 legacy=0x7f40c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6790/0 conn(0x7f40c8002840 legacy=0x7f40c8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.200+0000 7f40d97fa700 10 client.84107.objecter ms_handle_connect 0x7f40cc038360 2024-09-18T11:48:05.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40ee3ac700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:48:05.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f40c8007550 con 0x7f40c8006900 2024-09-18T11:48:05.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40ed3aa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6790/0 conn(0x7f40c8006900 legacy=0x7f40c8006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40d97fa700 10 client.84107.objecter ms_handle_connect 0x7f40c8006900 2024-09-18T11:48:05.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40d97fa700 10 client.84107.objecter resend_mon_ops 2024-09-18T11:48:05.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949306567 0 0) 0x7f40c8007550 con 0x7f40c8006900 2024-09-18T11:48:05.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40d97fa700 10 monclient(hunting): my global_id is 84113 2024-09-18T11:48:05.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c8006590 con 0x7f40c8006900 2024-09-18T11:48:05.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.212+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3554276819 0 0) 0x7f40c8006590 con 0x7f40c8006900 2024-09-18T11:48:05.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40c8007550 con 0x7f40c8006900 2024-09-18T11:48:05.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f40e4004da0 con 0x7f40c8006900 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 10 monclient: dump: 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2137407269 0 0) 0x7f40c8007550 con 0x7f40c8006900 2024-09-18T11:48:05.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f40c80087b0 con 0x7f40c8006900 2024-09-18T11:48:05.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 288911411) 0x7f40e4005030 con 0x7f40c8006900 2024-09-18T11:48:05.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:48:05.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:05.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6790/0 conn(0x7f40c8006900 legacy=0x7f40c8006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40d97fa700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6790/0 conn(0x7f40c8006900 legacy=0x7f40c8006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 20, 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:48:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:48:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:48:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:48:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:48:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:48:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:48:05.234 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:48:05.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 10 librados: watch_flush enter 2024-09-18T11:48:05.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 10 librados: watch_flush exit 2024-09-18T11:48:05.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 20 client.84107.objecter shutdown clearing up homeless session... 2024-09-18T11:48:05.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 10 client.84107.objecter successfully canceled tick 2024-09-18T11:48:05.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.129:6812/20806 conn(0x7f40cc038360 legacy=0x7f40cc03a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 10 monclient: shutdown 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 >> v1:172.21.15.112:6790/0 conn(0x7f40e80a0380 legacy=0x7f40e81465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40edbab700 1 -- 172.21.15.112:0/1559230994 reap_dead start 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40edbab700 1 -- 172.21.15.112:0/1559230994 reap_dead start 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 shutdown_connections 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 >> 172.21.15.112:0/1559230994 conn(0x7f40e8068100 msgr2=0x7f40e80985c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 shutdown_connections 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 1 -- 172.21.15.112:0/1559230994 wait complete. 2024-09-18T11:48:05.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.216+0000 7f40ee3ac700 1 librados: shutdown 2024-09-18T11:48:05.237 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:48:05.237 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:48:05.237 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient: get_monmap_and_config 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient: build_initial_monmap 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient: monmap: 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:05.344956+0000 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:05.344956+0000 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.345 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:05.346 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:05.346 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:05.346 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 1 Processor -- start 2024-09-18T11:48:05.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 1 -- start start 2024-09-18T11:48:05.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient: init 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient(hunting): picked mon.noname-b con 0x7f46040a0500 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient(hunting): picked mon.noname-c con 0x7f46040a0fe0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient(hunting): picked mon.noname-a con 0x7f4604097ac0 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46040a13e0 con 0x7f4604097ac0 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46040a3370 con 0x7f46040a0fe0 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46040a1660 con 0x7f46040a0500 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f4609d3d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f4604097ac0 0x7f460413ac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34048/0 (socket says 172.21.15.112:34048) 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f4609d3d700 1 -- 172.21.15.112:0/1063781891 learned_addr learned my addr 172.21.15.112:0/1063781891 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:05.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.340+0000 7f460b7a0700 10 monclient(hunting): authenticate will time out at 1057.187834s 2024-09-18T11:48:05.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f4609d3d700 1 -- 172.21.15.112:0/1063781891 >> v1:172.21.15.112:6789/0 conn(0x7f4604097ac0 legacy=0x7f460413ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f460953c700 1 -- 172.21.15.112:0/1063781891 >> v1:172.21.15.112:6790/0 conn(0x7f46040a0500 legacy=0x7f46040a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f4608d3b700 1 -- 172.21.15.112:0/1063781891 >> v1:172.21.15.129:6789/0 conn(0x7f46040a0fe0 legacy=0x7f4604137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619007619 0 0) 0x7f45fc0017d0 con 0x7f4604097ac0 2024-09-18T11:48:05.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): my global_id is 74535 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f0003570 con 0x7f4604097ac0 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137387776 0 0) 0x7f46040a1660 con 0x7f46040a0500 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): my global_id is 84116 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45fc0017d0 con 0x7f46040a0500 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907278505 0 0) 0x7f46040a3370 con 0x7f46040a0fe0 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient(hunting): my global_id is 17824 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46040a1660 con 0x7f46040a0fe0 2024-09-18T11:48:05.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4258593335 0 0) 0x7f45f0003570 con 0x7f4604097ac0 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 >> v1:172.21.15.112:6790/0 conn(0x7f46040a0500 legacy=0x7f46040a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 >> v1:172.21.15.129:6789/0 conn(0x7f46040a0fe0 legacy=0x7f4604137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: _finish_hunting 0 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 monclient: found mon.noname-a 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4604096a60 con 0x7f4604097ac0 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: _finish_auth 0 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f45fc001cb0 con 0x7f4604097ac0 2024-09-18T11:48:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: dump: 2024-09-18T11:48:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f460b7a0700 5 monclient: authenticate success, global_id 74535 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f460b7a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45fc001f40 con 0x7f4604097ac0 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 1 -- 172.21.15.112:0/1063781891 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f45fc004e10 con 0x7f4604097ac0 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f460b7a0700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f460b7a0700 4 set_mon_vals no callback set 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f45fbfff700 10 monclient: dump: 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.344+0000 7f460b7a0700 10 monclient: shutdown 2024-09-18T11:48:05.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- 172.21.15.112:0/1063781891 >> v1:172.21.15.112:6789/0 conn(0x7f4604097ac0 legacy=0x7f460413ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- 172.21.15.112:0/1063781891 shutdown_connections 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- 172.21.15.112:0/1063781891 >> 172.21.15.112:0/1063781891 conn(0x7f460406a9b0 msgr2=0x7f4604067680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- 172.21.15.112:0/1063781891 shutdown_connections 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- 172.21.15.112:0/1063781891 wait complete. 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient: build_initial_monmap 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient: monmap: 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 librados: starting msgr at 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 librados: starting objecter 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 Processor -- start 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- start start 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 librados: setting wanted keys 2024-09-18T11:48:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 librados: calling monclient init 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient: init 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient(hunting): picked mon.noname-b con 0x7f4604097ac0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient(hunting): picked mon.noname-a con 0x7f46040a0500 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient(hunting): picked mon.noname-c con 0x7f46040a0fe0 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46040a21e0 con 0x7f46040a0500 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4604098770 con 0x7f4604097ac0 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4604141e50 con 0x7f46040a0fe0 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f460b7a0700 10 monclient(hunting): authenticate will time out at 1057.193624s 2024-09-18T11:48:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f4609d3d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f46040a0fe0 0x7f4604149a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:32968/0 (socket says 172.21.15.112:32968) 2024-09-18T11:48:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f4608d3b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f46040a0500 0x7f46041355a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34062/0 (socket says 172.21.15.112:34062) 2024-09-18T11:48:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f4609d3d700 1 -- 172.21.15.112:0/454469497 learned_addr learned my addr 172.21.15.112:0/454469497 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:05.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.348+0000 7f4608d3b700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.112:6789/0 conn(0x7f46040a0500 legacy=0x7f46041355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 client.?.objecter ms_handle_connect 0x7f46040a0fe0 2024-09-18T11:48:05.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 client.?.objecter ms_handle_connect 0x7f46040a0500 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f4609d3d700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.112:6790/0 conn(0x7f46040a0fe0 legacy=0x7f4604149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460953c700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6789/0 conn(0x7f4604097ac0 legacy=0x7f4604141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 client.?.objecter ms_handle_connect 0x7f4604097ac0 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317337554 0 0) 0x7f46040a21e0 con 0x7f46040a0500 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): my global_id is 74538 2024-09-18T11:48:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e8004f10 con 0x7f46040a0500 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393403108 0 0) 0x7f4604141e50 con 0x7f46040a0fe0 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): my global_id is 84119 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46040a21e0 con 0x7f46040a0fe0 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274027967 0 0) 0x7f4604098770 con 0x7f4604097ac0 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): my global_id is 17827 2024-09-18T11:48:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4604141e50 con 0x7f4604097ac0 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2241930341 0 0) 0x7f46040a21e0 con 0x7f46040a0fe0 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4604098770 con 0x7f46040a0fe0 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f45fc001c30 con 0x7f46040a0fe0 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): dump: 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1548387732 0 0) 0x7f45e8004f10 con 0x7f46040a0500 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46040a21e0 con 0x7f46040a0500 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f45f4001c40 con 0x7f46040a0500 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient(hunting): dump: 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4049285610 0 0) 0x7f4604098770 con 0x7f46040a0fe0 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6789/0 conn(0x7f4604097ac0 legacy=0x7f4604141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.112:6789/0 conn(0x7f46040a0500 legacy=0x7f46041355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient: _finish_hunting 0 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 monclient: found mon.c 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46041217d0 con 0x7f46040a0fe0 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient: _finish_auth 0 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 5 monclient: authenticate success, global_id 84119 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 10 monclient: _renew_subs 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46041226c0 con 0x7f46040a0fe0 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 10 client.84119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 10 monclient: _renew_subs 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4604135cb0 con 0x7f46040a0fe0 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 1 librados: init done 2024-09-18T11:48:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45fc0053c0 con 0x7f46040a0fe0 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f45fc0056d0 con 0x7f46040a0fe0 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45fb7fe700 4 set_mon_vals no callback set 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 monclient: dump: 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.376 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f45e4006590 con 0x7f45e4002840 2024-09-18T11:48:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f45fc00ff50 con 0x7f46040a0fe0 2024-09-18T11:48:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(163..163 src has 1..163) v4 ==== 6119+0+0 (unknown 4251182424 0 0) 0x7f45fc044720 con 0x7f46040a0fe0 2024-09-18T11:48:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 10 client.84119.objecter ms_dispatch 0x7f4604000fd0 osd_map(163..163 src has 1..163) v4 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 3 client.84119.objecter handle_osd_map got epochs [163,163] > 0 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 3 client.84119.objecter handle_osd_map decoding full epoch 163 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.352+0000 7f45f97fa700 20 client.84119.objecter dump_active .. 0 homeless 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f4608d3b700 10 monclient: get_auth_request con 0x7f45e80385d0 auth_method 0 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 client.84119.objecter ms_handle_connect 0x7f45e4002840 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 client.84119.objecter resend_mon_ops 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f460953c700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6789/0 conn(0x7f45e4002840 legacy=0x7f45e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671163242 0 0) 0x7f45e4006590 con 0x7f45e4002840 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 monclient(hunting): my global_id is 17830 2024-09-18T11:48:05.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4604098770 con 0x7f45e4002840 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 client.84119.objecter ms_handle_connect 0x7f45e80385d0 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1377502281 0 0) 0x7f4604098770 con 0x7f45e4002840 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45e4006590 con 0x7f45e4002840 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4600001ac0 con 0x7f45e4002840 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 10 monclient: dump: 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1653179927 0 0) 0x7f45e4006590 con 0x7f45e4002840 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.356+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45e400a140 con 0x7f45e4002840 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.360+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f4600005810 con 0x7f45e4002840 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.360+0000 7f45f97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.360+0000 7f45f97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.360+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6789/0 conn(0x7f45e4002840 legacy=0x7f45e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.360+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6789/0 conn(0x7f45e4002840 legacy=0x7f45e4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.368+0000 7f460b7a0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:48:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.368+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f45e4009670 con 0x7f45e40090e0 2024-09-18T11:48:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 client.84119.objecter ms_handle_connect 0x7f45e40090e0 2024-09-18T11:48:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 client.84119.objecter resend_mon_ops 2024-09-18T11:48:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f460953c700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6789/0 conn(0x7f45e40090e0 legacy=0x7f45e4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117807565 0 0) 0x7f45e4009670 con 0x7f45e40090e0 2024-09-18T11:48:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 monclient(hunting): my global_id is 17833 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e4006590 con 0x7f45e40090e0 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2281749013 0 0) 0x7f45e4006590 con 0x7f45e40090e0 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45e4009670 con 0x7f45e40090e0 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4600002350 con 0x7f45e40090e0 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 monclient: dump: 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2797829863 0 0) 0x7f45e4009670 con 0x7f45e40090e0 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f45e4009930 con 0x7f45e40090e0 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 2807412054) 0x7f4600005a00 con 0x7f45e40090e0 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6789/0 conn(0x7f45e40090e0 legacy=0x7f45e4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.372+0000 7f45f97fa700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6789/0 conn(0x7f45e40090e0 legacy=0x7f45e4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:48:05.383 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 20, 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 6 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 10 librados: watch_flush enter 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 10 librados: watch_flush exit 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 20 client.84119.objecter shutdown clearing up homeless session... 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 10 client.84119.objecter successfully canceled tick 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.129:6812/20806 conn(0x7f45e80385d0 legacy=0x7f45e803aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 10 monclient: shutdown 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f4609d3d700 1 -- 172.21.15.112:0/454469497 reap_dead start 2024-09-18T11:48:05.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 >> v1:172.21.15.112:6790/0 conn(0x7f46040a0fe0 legacy=0x7f4604149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 shutdown_connections 2024-09-18T11:48:05.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 >> 172.21.15.112:0/454469497 conn(0x7f460406a9b0 msgr2=0x7f4604098a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:05.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 shutdown_connections 2024-09-18T11:48:05.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 1 -- 172.21.15.112:0/454469497 wait complete. 2024-09-18T11:48:05.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.376+0000 7f460b7a0700 1 librados: shutdown 2024-09-18T11:48:05.397 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:48:05.397 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T11:48:05.397 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T11:48:05.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient: get_monmap_and_config 2024-09-18T11:48:05.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient: build_initial_monmap 2024-09-18T11:48:05.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient: monmap: 2024-09-18T11:48:05.513 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:05.513 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.514 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:05.512944+0000 2024-09-18T11:48:05.514 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:05.512944+0000 2024-09-18T11:48:05.514 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:05.514 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.514 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:05.516 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:05.516 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:05.516 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 1 Processor -- start 2024-09-18T11:48:05.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 1 -- start start 2024-09-18T11:48:05.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient: init 2024-09-18T11:48:05.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:05.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient(hunting): picked mon.noname-a con 0x7fe0e406aa40 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient(hunting): picked mon.noname-c con 0x7fe0e40a1e80 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient(hunting): picked mon.noname-b con 0x7fe0e41386e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e4097270 con 0x7fe0e406aa40 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e406ae10 con 0x7fe0e40a1e80 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e4097550 con 0x7fe0e41386e0 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0eb1af700 10 monclient(hunting): authenticate will time out at 1057.354280s 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0e8f4b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe0e406aa40 0x7fe0e40a0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34066/0 (socket says 172.21.15.112:34066) 2024-09-18T11:48:05.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.508+0000 7fe0e8f4b700 1 -- 172.21.15.112:0/1916346136 learned_addr learned my addr 172.21.15.112:0/1916346136 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e8f4b700 1 -- 172.21.15.112:0/1916346136 >> v1:172.21.15.112:6789/0 conn(0x7fe0e406aa40 legacy=0x7fe0e40a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e974c700 1 -- 172.21.15.112:0/1916346136 >> v1:172.21.15.112:6790/0 conn(0x7fe0e41386e0 legacy=0x7fe0e413aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e3fff700 1 -- 172.21.15.112:0/1916346136 >> v1:172.21.15.129:6789/0 conn(0x7fe0e40a1e80 legacy=0x7fe0e40a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955305772 0 0) 0x7fe0e4097270 con 0x7fe0e406aa40 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): my global_id is 74544 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0cc003570 con 0x7fe0e406aa40 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981699169 0 0) 0x7fe0e4097550 con 0x7fe0e41386e0 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): my global_id is 84128 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e4097270 con 0x7fe0e41386e0 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936648146 0 0) 0x7fe0e406ae10 con 0x7fe0e40a1e80 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient(hunting): my global_id is 17848 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e4097550 con 0x7fe0e40a1e80 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1700349902 0 0) 0x7fe0cc003570 con 0x7fe0e406aa40 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 >> v1:172.21.15.112:6790/0 conn(0x7fe0e41386e0 legacy=0x7fe0e413aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 >> v1:172.21.15.129:6789/0 conn(0x7fe0e40a1e80 legacy=0x7fe0e40a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: _finish_hunting 0 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 monclient: found mon.noname-a 2024-09-18T11:48:05.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e4096a60 con 0x7fe0e406aa40 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: _finish_auth 0 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0d4001940 con 0x7fe0e406aa40 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: dump: 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.524 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 5 monclient: authenticate success, global_id 74544 2024-09-18T11:48:05.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0d4001bd0 con 0x7fe0e406aa40 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 1 -- 172.21.15.112:0/1916346136 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0d4004a90 con 0x7fe0e406aa40 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 4 set_mon_vals no callback set 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0e37fe700 10 monclient: dump: 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.526 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 10 monclient: shutdown 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 1 -- 172.21.15.112:0/1916346136 >> v1:172.21.15.112:6789/0 conn(0x7fe0e406aa40 legacy=0x7fe0e40a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 1 -- 172.21.15.112:0/1916346136 shutdown_connections 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 1 -- 172.21.15.112:0/1916346136 >> 172.21.15.112:0/1916346136 conn(0x7fe0e40a5d10 msgr2=0x7fe0e40a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 1 -- 172.21.15.112:0/1916346136 shutdown_connections 2024-09-18T11:48:05.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 1 -- 172.21.15.112:0/1916346136 wait complete. 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 10 monclient: build_initial_monmap 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.512+0000 7fe0eb1af700 10 monclient: monmap: 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:05.528 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 librados: starting msgr at 2024-09-18T11:48:05.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 librados: starting objecter 2024-09-18T11:48:05.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 Processor -- start 2024-09-18T11:48:05.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 -- start start 2024-09-18T11:48:05.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 librados: setting wanted keys 2024-09-18T11:48:05.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 librados: calling monclient init 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 10 monclient: init 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 10 monclient(hunting): picked mon.noname-a con 0x7fe0e406aa40 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 10 monclient(hunting): picked mon.noname-c con 0x7fe0e40a1e80 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 10 monclient(hunting): picked mon.noname-b con 0x7fe0e41386e0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e40a22f0 con 0x7fe0e406aa40 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0e8f4b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe0e406aa40 0x7fe0e4135b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34082/0 (socket says 172.21.15.112:34082) 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0e8f4b700 1 -- 172.21.15.112:0/2138582402 learned_addr learned my addr 172.21.15.112:0/2138582402 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e41314e0 con 0x7fe0e41386e0 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e4097d10 con 0x7fe0e40a1e80 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.516+0000 7fe0eb1af700 10 monclient(hunting): authenticate will time out at 1057.362096s 2024-09-18T11:48:05.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e8f4b700 1 -- 172.21.15.112:0/2138582402 >> v1:172.21.15.112:6789/0 conn(0x7fe0e406aa40 legacy=0x7fe0e4135b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 client.?.objecter ms_handle_connect 0x7fe0e406aa40 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 client.?.objecter ms_handle_connect 0x7fe0e40a1e80 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e3fff700 1 -- 172.21.15.112:0/2138582402 >> v1:172.21.15.112:6790/0 conn(0x7fe0e40a1e80 legacy=0x7fe0e4136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106914147 0 0) 0x7fe0e40a22f0 con 0x7fe0e406aa40 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): my global_id is 74547 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d0004f10 con 0x7fe0e406aa40 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 client.?.objecter ms_handle_connect 0x7fe0e41386e0 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e974c700 1 -- 172.21.15.112:0/2138582402 >> v1:172.21.15.129:6789/0 conn(0x7fe0e41386e0 legacy=0x7fe0e414a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481069017 0 0) 0x7fe0e41314e0 con 0x7fe0e41386e0 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): my global_id is 17851 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e40a22f0 con 0x7fe0e41386e0 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 826724844 0 0) 0x7fe0d0004f10 con 0x7fe0e406aa40 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0e41314e0 con 0x7fe0e406aa40 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0d40049d0 con 0x7fe0e406aa40 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): dump: 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.536 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3063127060 0 0) 0x7fe0e40a22f0 con 0x7fe0e41386e0 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0d0004f10 con 0x7fe0e41386e0 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0dc001ff0 con 0x7fe0e41386e0 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient(hunting): dump: 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 642855181 0 0) 0x7fe0e41314e0 con 0x7fe0e406aa40 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 >> v1:172.21.15.112:6790/0 conn(0x7fe0e40a1e80 legacy=0x7fe0e4136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 >> v1:172.21.15.129:6789/0 conn(0x7fe0e41386e0 legacy=0x7fe0e414a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 monclient: found mon.a 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e4136970 con 0x7fe0e406aa40 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient: _finish_auth 0 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 5 monclient: authenticate success, global_id 74547 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 10 monclient: _renew_subs 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e40981e0 con 0x7fe0e406aa40 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 10 client.74547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 10 monclient: _renew_subs 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e409eab0 con 0x7fe0e406aa40 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0d4004f40 con 0x7fe0e406aa40 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 1 librados: init done 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0d4005250 con 0x7fe0e406aa40 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 monclient: dump: 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.540 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e2ffd700 4 set_mon_vals no callback set 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e413c120 con 0x7fe0e406aa40 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe0d400fad0 con 0x7fe0e406aa40 2024-09-18T11:48:05.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(164..164 src has 1..164) v4 ==== 5615+0+0 (unknown 2042975791 0 0) 0x7fe0d40440f0 con 0x7fe0e406aa40 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 10 client.74547.objecter ms_dispatch 0x7fe0e4000fd0 osd_map(164..164 src has 1..164) v4 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 3 client.74547.objecter handle_osd_map got epochs [164,164] > 0 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 3 client.74547.objecter handle_osd_map decoding full epoch 164 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.520+0000 7fe0e0ff9700 20 client.74547.objecter dump_active .. 0 homeless 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.524+0000 7fe0e3fff700 10 monclient: get_auth_request con 0x7fe0d0038300 auth_method 0 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.524+0000 7fe0e0ff9700 10 client.74547.objecter ms_handle_connect 0x7fe0d0038300 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.536+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fe0d4014c40 con 0x7fe0e406aa40 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.536+0000 7fe0e0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:05.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.536+0000 7fe0e0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:05.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.764+0000 7fe0eb1af700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T11:48:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.764+0000 7fe0eb1af700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T11:48:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.764+0000 7fe0eb1af700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.764+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe0e4099ed0 con 0x7fe0e406aa40 2024-09-18T11:48:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.764+0000 7fe0e0ff9700 1 -- 172.21.15.112:0/2138582402 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fe0d4014640 con 0x7fe0e406aa40 2024-09-18T11:48:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.764+0000 7fe0e0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T11:48:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.764+0000 7fe0e0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:05.772 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.764+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.764+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=1984075649,config=3140514009,health=3024419634,kv=1310296871,logm=2689182825}) 2024-09-18T11:48:05.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.764+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=7,logm=64} crc {auth=2777102898,config=3140514009,health=67172602,kv=1310296871,logm=3740498129}) 2024-09-18T11:48:05.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.764+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.764+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=1984075649,config=3140514009,health=3024419634,kv=1310296871,logm=2689182825}) 2024-09-18T11:48:05.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.764+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=8,logm=66} crc {auth=3265809436,config=3140514009,health=3861840688,kv=1310296871,logm=3936718186}) 2024-09-18T11:48:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 10 librados: watch_flush enter 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 10 librados: watch_flush exit 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 20 client.74547.objecter shutdown clearing up homeless session... 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 10 client.74547.objecter successfully canceled tick 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 >> v1:172.21.15.129:6812/20806 conn(0x7fe0d0038300 legacy=0x7fe0d003a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 10 monclient: shutdown 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 >> v1:172.21.15.112:6789/0 conn(0x7fe0e406aa40 legacy=0x7fe0e4135b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 shutdown_connections 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 >> 172.21.15.112:0/2138582402 conn(0x7fe0e40a5d10 msgr2=0x7fe0e4137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 shutdown_connections 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 1 -- 172.21.15.112:0/2138582402 wait complete. 2024-09-18T11:48:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:05.768+0000 7fe0eb1af700 1 librados: shutdown 2024-09-18T11:48:05.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.768+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.768+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436737142}) 2024-09-18T11:48:05.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.768+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174795825}) 2024-09-18T11:48:05.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.768+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.768+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436737142}) 2024-09-18T11:48:05.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.768+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666122995}) 2024-09-18T11:48:05.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537229199}) 2024-09-18T11:48:05.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=857489635}) 2024-09-18T11:48:05.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537229199}) 2024-09-18T11:48:05.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1769141788}) 2024-09-18T11:48:05.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214739604}) 2024-09-18T11:48:05.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1403814328}) 2024-09-18T11:48:05.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214739604}) 2024-09-18T11:48:05.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.772+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1844276972}) 2024-09-18T11:48:05.781 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.781 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154398763}) 2024-09-18T11:48:05.781 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2239387628}) 2024-09-18T11:48:05.782 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.782 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154398763}) 2024-09-18T11:48:05.782 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149941002}) 2024-09-18T11:48:05.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722679563}) 2024-09-18T11:48:05.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1557050834}) 2024-09-18T11:48:05.784 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.784 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722679563}) 2024-09-18T11:48:05.784 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=544930552}) 2024-09-18T11:48:05.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=311097502}) 2024-09-18T11:48:05.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1647931194}) 2024-09-18T11:48:05.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=311097502}) 2024-09-18T11:48:05.786 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.776+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288190873}) 2024-09-18T11:48:05.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T11:48:05.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003682315}) 2024-09-18T11:48:05.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2125968034}) 2024-09-18T11:48:05.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003682315}) 2024-09-18T11:48:05.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179187515}) 2024-09-18T11:48:05.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713109171}) 2024-09-18T11:48:05.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052616663}) 2024-09-18T11:48:05.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713109171}) 2024-09-18T11:48:05.799 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3789543744}) 2024-09-18T11:48:05.799 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4118606067}) 2024-09-18T11:48:05.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252033022}) 2024-09-18T11:48:05.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4118606067}) 2024-09-18T11:48:05.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.792+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780784157}) 2024-09-18T11:48:05.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377073974}) 2024-09-18T11:48:05.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1437788047}) 2024-09-18T11:48:05.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377073974}) 2024-09-18T11:48:05.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135096106}) 2024-09-18T11:48:05.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766806457}) 2024-09-18T11:48:05.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2915588069}) 2024-09-18T11:48:05.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766806457}) 2024-09-18T11:48:05.803 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1624971869}) 2024-09-18T11:48:05.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=620232519}) 2024-09-18T11:48:05.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1351685830}) 2024-09-18T11:48:05.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=620232519}) 2024-09-18T11:48:05.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.796+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2523545805}) 2024-09-18T11:48:05.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556889501}) 2024-09-18T11:48:05.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3986608459}) 2024-09-18T11:48:05.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556889501}) 2024-09-18T11:48:05.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1249224525}) 2024-09-18T11:48:05.806 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351689088}) 2024-09-18T11:48:05.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1622087705}) 2024-09-18T11:48:05.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351689088}) 2024-09-18T11:48:05.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831542813}) 2024-09-18T11:48:05.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614058729}) 2024-09-18T11:48:05.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987332712}) 2024-09-18T11:48:05.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.809 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614058729}) 2024-09-18T11:48:05.809 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.800+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429318040}) 2024-09-18T11:48:05.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154662062}) 2024-09-18T11:48:05.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=692519325}) 2024-09-18T11:48:05.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154662062}) 2024-09-18T11:48:05.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2620545782}) 2024-09-18T11:48:05.811 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.811 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701378277}) 2024-09-18T11:48:05.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550937824}) 2024-09-18T11:48:05.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701378277}) 2024-09-18T11:48:05.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.804+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=917971336}) 2024-09-18T11:48:05.813 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.813 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025650937}) 2024-09-18T11:48:05.814 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350367918}) 2024-09-18T11:48:05.814 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.814 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025650937}) 2024-09-18T11:48:05.814 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2659103380}) 2024-09-18T11:48:05.814 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640240}) 2024-09-18T11:48:05.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1458065697}) 2024-09-18T11:48:05.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640240}) 2024-09-18T11:48:05.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=150092801}) 2024-09-18T11:48:05.816 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20} crc {logm=3599372832,mdsmap=1531151339,mgr=3772577836,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=884570931,mon_config_key=212001074,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=3736642984,osd_snap=1890956289}) 2024-09-18T11:48:05.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=43,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=18} crc {logm=481351475,mdsmap=1531151339,mgr=3797927048,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=884570931,mon_config_key=868353578,monmap=1725997317,osd_metadata=1765517589,osd_pg_creating=3736642984,osd_snap=4172013305}) 2024-09-18T11:48:05.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20} crc {logm=3599372832,mdsmap=1531151339,mgr=3772577836,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=884570931,mon_config_key=212001074,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=3736642984,osd_snap=1890956289}) 2024-09-18T11:48:05.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.808+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=16} crc {logm=1149763015,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=3903824952,mon_config_key=868353578,monmap=1725997317,osd_metadata=78095468,osd_pg_creating=3736642984,osd_snap=786705438}) 2024-09-18T11:48:05.818 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.818 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=482010601,osdmap=2651067796}) 2024-09-18T11:48:05.818 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=482010601,osdmap=2958303092}) 2024-09-18T11:48:05.818 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.818 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=482010601,osdmap=2651067796}) 2024-09-18T11:48:05.818 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=482010602,osdmap=3624042221}) 2024-09-18T11:48:05.820 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.820 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3332791086}) 2024-09-18T11:48:05.820 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2542076879}) 2024-09-18T11:48:05.820 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.820 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3332791086}) 2024-09-18T11:48:05.821 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.812+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1508007100}) 2024-09-18T11:48:05.827 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.820+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.827 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.820+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=1540814754}) 2024-09-18T11:48:05.827 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.820+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=62} crc {osdmap=2173643192}) 2024-09-18T11:48:05.828 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.820+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:48:05.828 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.820+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=1540814754}) 2024-09-18T11:48:05.828 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:48:05.820+0000 7f9346370700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=60} crc {osdmap=3411811419}) 2024-09-18T11:48:06.787 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:48:06.788 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:48:06.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient: get_monmap_and_config 2024-09-18T11:48:06.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient: build_initial_monmap 2024-09-18T11:48:06.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient: monmap: 2024-09-18T11:48:06.901 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:06.901 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:06.902 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:06.901144+0000 2024-09-18T11:48:06.902 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:06.901144+0000 2024-09-18T11:48:06.902 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:06.902 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:06.902 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:06.902 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:06.902 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:06.902 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 1 Processor -- start 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 1 -- start start 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient: init 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient(hunting): picked mon.noname-b con 0x7f3a900a0570 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient(hunting): picked mon.noname-a con 0x7f3a900a1050 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient(hunting): picked mon.noname-c con 0x7f3a90097a50 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a90097070 con 0x7f3a900a1050 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a90096d90 con 0x7f3a90097a50 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a90097e50 con 0x7f3a900a0570 2024-09-18T11:48:06.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:06.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a97f69700 10 monclient(hunting): authenticate will time out at 1058.742516s 2024-09-18T11:48:06.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a95504700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3a900a1050 0x7f3a90137780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34086/0 (socket says 172.21.15.112:34086) 2024-09-18T11:48:06.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a95d05700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3a900a0570 0x7f3a900a0940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33012/0 (socket says 172.21.15.112:33012) 2024-09-18T11:48:06.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.896+0000 7f3a95504700 1 -- 172.21.15.112:0/3027748570 learned_addr learned my addr 172.21.15.112:0/3027748570 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a95504700 1 -- 172.21.15.112:0/3027748570 >> v1:172.21.15.112:6789/0 conn(0x7f3a900a1050 legacy=0x7f3a90137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a95d05700 1 -- 172.21.15.112:0/3027748570 >> v1:172.21.15.112:6790/0 conn(0x7f3a900a0570 legacy=0x7f3a900a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701450860 0 0) 0x7f3a90097070 con 0x7f3a900a1050 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient(hunting): my global_id is 74550 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a78003570 con 0x7f3a900a1050 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124532658 0 0) 0x7f3a90097e50 con 0x7f3a900a0570 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:06.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient(hunting): my global_id is 84131 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a90097070 con 0x7f3a900a0570 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a96506700 1 -- 172.21.15.112:0/3027748570 >> v1:172.21.15.129:6789/0 conn(0x7f3a90097a50 legacy=0x7f3a9013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4093232544 0 0) 0x7f3a90097070 con 0x7f3a900a0570 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 >> v1:172.21.15.129:6789/0 conn(0x7f3a90097a50 legacy=0x7f3a9013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 >> v1:172.21.15.112:6789/0 conn(0x7f3a900a1050 legacy=0x7f3a90137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: _finish_hunting 0 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 monclient: found mon.noname-b 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a900a2050 con 0x7f3a900a0570 2024-09-18T11:48:06.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: _finish_auth 0 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3a80001960 con 0x7f3a900a0570 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 5 monclient: authenticate success, global_id 84131 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: dump: 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.909 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a80001b40 con 0x7f3a900a0570 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 1 -- 172.21.15.112:0/3027748570 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3a80004830 con 0x7f3a900a0570 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 4 set_mon_vals no callback set 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:06.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a94d03700 10 monclient: dump: 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:06.911 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:06.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 10 monclient: shutdown 2024-09-18T11:48:06.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:06.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 1 -- 172.21.15.112:0/3027748570 >> v1:172.21.15.112:6790/0 conn(0x7f3a900a0570 legacy=0x7f3a900a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:06.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 1 -- 172.21.15.112:0/3027748570 shutdown_connections 2024-09-18T11:48:06.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 1 -- 172.21.15.112:0/3027748570 >> 172.21.15.112:0/3027748570 conn(0x7f3a90067680 msgr2=0x7f3a90067a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:06.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 1 -- 172.21.15.112:0/3027748570 shutdown_connections 2024-09-18T11:48:06.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.900+0000 7f3a97f69700 1 -- 172.21.15.112:0/3027748570 wait complete. 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient: build_initial_monmap 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient: monmap: 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 librados: starting msgr at 2024-09-18T11:48:06.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 librados: starting objecter 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 Processor -- start 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 -- start start 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 librados: setting wanted keys 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 librados: calling monclient init 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient: init 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient(hunting): picked mon.noname-c con 0x7f3a90097a50 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient(hunting): picked mon.noname-a con 0x7f3a900a0570 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient(hunting): picked mon.noname-b con 0x7f3a900a1050 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a900983a0 con 0x7f3a900a0570 2024-09-18T11:48:06.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a901228c0 con 0x7f3a900a1050 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a901225e0 con 0x7f3a90097a50 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a97f69700 10 monclient(hunting): authenticate will time out at 1058.750231s 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a95504700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3a900a0570 0x7f3a90146960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34094/0 (socket says 172.21.15.112:34094) 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.904+0000 7f3a95504700 1 -- 172.21.15.112:0/2333191404 learned_addr learned my addr 172.21.15.112:0/2333191404 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a95504700 1 -- 172.21.15.112:0/2333191404 >> v1:172.21.15.112:6789/0 conn(0x7f3a900a0570 legacy=0x7f3a90146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 client.?.objecter ms_handle_connect 0x7f3a900a0570 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 client.?.objecter ms_handle_connect 0x7f3a90097a50 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a95d05700 1 -- 172.21.15.112:0/2333191404 >> v1:172.21.15.112:6790/0 conn(0x7f3a90097a50 legacy=0x7f3a901354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734301816 0 0) 0x7f3a900983a0 con 0x7f3a900a0570 2024-09-18T11:48:06.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): my global_id is 74553 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a96506700 1 -- 172.21.15.112:0/2333191404 >> v1:172.21.15.129:6789/0 conn(0x7f3a900a1050 legacy=0x7f3a9014a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a7c004f10 con 0x7f3a900a0570 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 client.?.objecter ms_handle_connect 0x7f3a900a1050 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618933076 0 0) 0x7f3a901225e0 con 0x7f3a90097a50 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): my global_id is 84134 2024-09-18T11:48:06.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a900983a0 con 0x7f3a90097a50 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073782820 0 0) 0x7f3a901228c0 con 0x7f3a900a1050 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): my global_id is 17857 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a901225e0 con 0x7f3a900a1050 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3494862160 0 0) 0x7f3a7c004f10 con 0x7f3a900a0570 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a901228c0 con 0x7f3a900a0570 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3a8c001f90 con 0x7f3a900a0570 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): dump: 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:06.921 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748832215 0 0) 0x7f3a900983a0 con 0x7f3a90097a50 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a7c004f10 con 0x7f3a90097a50 2024-09-18T11:48:06.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3a80001360 con 0x7f3a90097a50 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): dump: 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:06.923 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1206610104 0 0) 0x7f3a901225e0 con 0x7f3a900a1050 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a900983a0 con 0x7f3a900a1050 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3a88001a40 con 0x7f3a900a1050 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient(hunting): dump: 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:06.924 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3416954983 0 0) 0x7f3a901228c0 con 0x7f3a900a0570 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 >> v1:172.21.15.112:6790/0 conn(0x7f3a90097a50 legacy=0x7f3a901354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 >> v1:172.21.15.129:6789/0 conn(0x7f3a900a1050 legacy=0x7f3a9014a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 monclient: found mon.a 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a90120ab0 con 0x7f3a900a0570 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient: _finish_auth 0 2024-09-18T11:48:06.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 5 monclient: authenticate success, global_id 74553 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 10 monclient: _renew_subs 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a901423e0 con 0x7f3a900a0570 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 10 client.74553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 10 monclient: _renew_subs 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a90135bd0 con 0x7f3a900a0570 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 1 librados: init done 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a8c004e60 con 0x7f3a900a0570 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3a8c0051f0 con 0x7f3a900a0570 2024-09-18T11:48:06.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a87fff700 4 set_mon_vals no callback set 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a85ffb700 10 monclient: dump: 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:06.927 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:06.928 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:06.928 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:06.928 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:06.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:06.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:06.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:06.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.908+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a700050b0 con 0x7f3a900a0570 2024-09-18T11:48:06.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.912+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f3a8c005350 con 0x7f3a900a0570 2024-09-18T11:48:06.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.912+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(165..165 src has 1..165) v4 ==== 6297+0+0 (unknown 596103688 0 0) 0x7f3a8c0446f0 con 0x7f3a900a0570 2024-09-18T11:48:06.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.912+0000 7f3a85ffb700 10 client.74553.objecter ms_dispatch 0x7f3a90000fd0 osd_map(165..165 src has 1..165) v4 2024-09-18T11:48:06.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.912+0000 7f3a85ffb700 3 client.74553.objecter handle_osd_map got epochs [165,165] > 0 2024-09-18T11:48:06.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.912+0000 7f3a85ffb700 3 client.74553.objecter handle_osd_map decoding full epoch 165 2024-09-18T11:48:06.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.912+0000 7f3a95d05700 10 monclient: get_auth_request con 0x7f3a7c0385b0 auth_method 0 2024-09-18T11:48:06.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.912+0000 7f3a85ffb700 20 client.74553.objecter dump_active .. 0 homeless 2024-09-18T11:48:06.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.912+0000 7f3a85ffb700 10 client.74553.objecter ms_handle_connect 0x7f3a7c0385b0 2024-09-18T11:48:06.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.920+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f3a8c014ed0 con 0x7f3a900a0570 2024-09-18T11:48:06.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.920+0000 7f3a85ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:06.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:06.920+0000 7f3a85ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:07.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.160+0000 7f3a97f69700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:48:07.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.160+0000 7f3a97f69700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:07.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.160+0000 7f3a97f69700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.160+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3a700058e0 con 0x7f3a900a0570 2024-09-18T11:48:07.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.160+0000 7f3a85ffb700 1 -- 172.21.15.112:0/2333191404 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2458706117) 0x7f3a8c0148d0 con 0x7f3a900a0570 2024-09-18T11:48:07.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.160+0000 7f3a85ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:07.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.160+0000 7f3a85ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:07.168 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:48:07.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 10 librados: watch_flush enter 2024-09-18T11:48:07.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 10 librados: watch_flush exit 2024-09-18T11:48:07.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 20 client.74553.objecter shutdown clearing up homeless session... 2024-09-18T11:48:07.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 10 client.74553.objecter successfully canceled tick 2024-09-18T11:48:07.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 >> v1:172.21.15.129:6812/20806 conn(0x7f3a7c0385b0 legacy=0x7f3a7c03aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 10 monclient: shutdown 2024-09-18T11:48:07.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:07.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 >> v1:172.21.15.112:6789/0 conn(0x7f3a900a0570 legacy=0x7f3a90146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 shutdown_connections 2024-09-18T11:48:07.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 >> 172.21.15.112:0/2333191404 conn(0x7f3a90067680 msgr2=0x7f3a900a5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:07.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 shutdown_connections 2024-09-18T11:48:07.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 1 -- 172.21.15.112:0/2333191404 wait complete. 2024-09-18T11:48:07.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.164+0000 7f3a97f69700 1 librados: shutdown 2024-09-18T11:48:07.181 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:48:07.181 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T11:48:07.181 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:48:07.182 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:48:07.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient: get_monmap_and_config 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient: build_initial_monmap 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient: monmap: 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:07.281401+0000 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:07.281401+0000 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:07.282 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 1 Processor -- start 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 1 -- start start 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient: init 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient(hunting): picked mon.noname-b con 0x7f9b600a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient(hunting): picked mon.noname-a con 0x7f9b600a0e60 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient(hunting): picked mon.noname-c con 0x7f9b60097970 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b60097550 con 0x7f9b600a0e60 2024-09-18T11:48:07.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b60096cb0 con 0x7f9b60097970 2024-09-18T11:48:07.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b600a1440 con 0x7f9b600a0380 2024-09-18T11:48:07.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:07.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b64aa3700 10 monclient(hunting): authenticate will time out at 1059.123200s 2024-09-18T11:48:07.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b56ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9b600a0e60 0x7f9b60137390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34108/0 (socket says 172.21.15.112:34108) 2024-09-18T11:48:07.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.276+0000 7f9b56ffd700 1 -- 172.21.15.112:0/3339761161 learned_addr learned my addr 172.21.15.112:0/3339761161 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:07.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b56ffd700 1 -- 172.21.15.112:0/3339761161 >> v1:172.21.15.112:6789/0 conn(0x7f9b600a0e60 legacy=0x7f9b60137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5f7fe700 1 -- 172.21.15.112:0/3339761161 >> v1:172.21.15.112:6790/0 conn(0x7f9b600a0380 legacy=0x7f9b600a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5ffff700 1 -- 172.21.15.112:0/3339761161 >> v1:172.21.15.129:6789/0 conn(0x7f9b60097970 legacy=0x7f9b6013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653421108 0 0) 0x7f9b60097550 con 0x7f9b600a0e60 2024-09-18T11:48:07.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): my global_id is 74556 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b4c003570 con 0x7f9b600a0e60 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982451091 0 0) 0x7f9b600a1440 con 0x7f9b600a0380 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): my global_id is 84137 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b60097550 con 0x7f9b600a0380 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400132382 0 0) 0x7f9b60096cb0 con 0x7f9b60097970 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient(hunting): my global_id is 17860 2024-09-18T11:48:07.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b600a1440 con 0x7f9b60097970 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4139573584 0 0) 0x7f9b4c003570 con 0x7f9b600a0e60 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 >> v1:172.21.15.112:6790/0 conn(0x7f9b600a0380 legacy=0x7f9b600a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 >> v1:172.21.15.129:6789/0 conn(0x7f9b60097970 legacy=0x7f9b6013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: _finish_hunting 0 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 monclient: found mon.noname-a 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b600a1de0 con 0x7f9b600a0e60 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: _finish_auth 0 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9b48001940 con 0x7f9b600a0e60 2024-09-18T11:48:07.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 5 monclient: authenticate success, global_id 74556 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: dump: 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.291 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b48001bd0 con 0x7f9b600a0e60 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 1 -- 172.21.15.112:0/3339761161 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9b48004a90 con 0x7f9b600a0e60 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 4 set_mon_vals no callback set 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b5effd700 10 monclient: dump: 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.292 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 10 monclient: shutdown 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 1 -- 172.21.15.112:0/3339761161 >> v1:172.21.15.112:6789/0 conn(0x7f9b600a0e60 legacy=0x7f9b60137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 1 -- 172.21.15.112:0/3339761161 shutdown_connections 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 1 -- 172.21.15.112:0/3339761161 >> 172.21.15.112:0/3339761161 conn(0x7f9b60068100 msgr2=0x7f9b600a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 1 -- 172.21.15.112:0/3339761161 shutdown_connections 2024-09-18T11:48:07.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.280+0000 7f9b64aa3700 1 -- 172.21.15.112:0/3339761161 wait complete. 2024-09-18T11:48:07.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient: build_initial_monmap 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient: monmap: 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 librados: starting msgr at 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 librados: starting objecter 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 Processor -- start 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 -- start start 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 librados: setting wanted keys 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 librados: calling monclient init 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient: init 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:48:07.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient(hunting): picked mon.noname-b con 0x7f9b60097970 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient(hunting): picked mon.noname-c con 0x7f9b600a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient(hunting): picked mon.noname-a con 0x7f9b600a0e60 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b60120ec0 con 0x7f9b600a0e60 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b600a2050 con 0x7f9b60097970 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b600984b0 con 0x7f9b600a0380 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b64aa3700 10 monclient(hunting): authenticate will time out at 1059.130185s 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b56ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f9b600a0380 0x7f9b601352e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33020/0 (socket says 172.21.15.112:33020) 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.284+0000 7f9b56ffd700 1 -- 172.21.15.112:0/47627074 learned_addr learned my addr 172.21.15.112:0/47627074 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b56ffd700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6790/0 conn(0x7f9b600a0380 legacy=0x7f9b601352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 client.?.objecter ms_handle_connect 0x7f9b600a0380 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 client.?.objecter ms_handle_connect 0x7f9b600a0e60 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b5ffff700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6789/0 conn(0x7f9b600a0e60 legacy=0x7f9b60149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 client.?.objecter ms_handle_connect 0x7f9b60097970 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b5f7fe700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.129:6789/0 conn(0x7f9b60097970 legacy=0x7f9b601414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615547417 0 0) 0x7f9b600984b0 con 0x7f9b600a0380 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): my global_id is 84140 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b40004f10 con 0x7f9b600a0380 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241779680 0 0) 0x7f9b60120ec0 con 0x7f9b600a0e60 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): my global_id is 74559 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b600984b0 con 0x7f9b600a0e60 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222397868 0 0) 0x7f9b600a2050 con 0x7f9b60097970 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): my global_id is 17863 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b60120ec0 con 0x7f9b60097970 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2006705677 0 0) 0x7f9b600984b0 con 0x7f9b600a0e60 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b600a2050 con 0x7f9b600a0e60 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1504235774 0 0) 0x7f9b40004f10 con 0x7f9b600a0380 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b600984b0 con 0x7f9b600a0380 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9b58001fc0 con 0x7f9b600a0e60 2024-09-18T11:48:07.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): dump: 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9b480049d0 con 0x7f9b600a0380 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): dump: 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.305 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2067864226 0 0) 0x7f9b60120ec0 con 0x7f9b60097970 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b40004f10 con 0x7f9b60097970 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9b500019c0 con 0x7f9b60097970 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient(hunting): dump: 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.306 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4272326799 0 0) 0x7f9b600a2050 con 0x7f9b600a0e60 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6790/0 conn(0x7f9b600a0380 legacy=0x7f9b601352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.129:6789/0 conn(0x7f9b60097970 legacy=0x7f9b601414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient: _finish_hunting 0 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 monclient: found mon.a 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b60122400 con 0x7f9b600a0e60 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient: _finish_auth 0 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 5 monclient: authenticate success, global_id 74559 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 10 monclient: _renew_subs 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b60120940 con 0x7f9b600a0e60 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 10 client.74559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 10 monclient: _renew_subs 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b601359f0 con 0x7f9b600a0e60 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 1 librados: init done 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b58002530 con 0x7f9b600a0e60 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:07.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9b580028c0 con 0x7f9b600a0e60 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b5e7fc700 4 set_mon_vals no callback set 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b57fff700 10 monclient: dump: 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.309 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.288+0000 7f9b64aa3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9b44006590 con 0x7f9b44002840 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f9b58002a20 con 0x7f9b600a0e60 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(166..166 src has 1..166) v4 ==== 6835+0+0 (unknown 3653208750 0 0) 0x7f9b580446a0 con 0x7f9b600a0e60 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 client.74559.objecter ms_dispatch 0x7f9b60000fd0 osd_map(166..166 src has 1..166) v4 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 3 client.74559.objecter handle_osd_map got epochs [166,166] > 0 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 3 client.74559.objecter handle_osd_map decoding full epoch 166 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 20 client.74559.objecter dump_active .. 0 homeless 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b56ffd700 10 monclient: get_auth_request con 0x7f9b40038610 auth_method 0 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b5f7fe700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6789/0 conn(0x7f9b44002840 legacy=0x7f9b44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 client.74559.objecter ms_handle_connect 0x7f9b44002840 2024-09-18T11:48:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 client.74559.objecter resend_mon_ops 2024-09-18T11:48:07.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478404719 0 0) 0x7f9b44006590 con 0x7f9b44002840 2024-09-18T11:48:07.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 monclient(hunting): my global_id is 74562 2024-09-18T11:48:07.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b600a2050 con 0x7f9b44002840 2024-09-18T11:48:07.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3833856095 0 0) 0x7f9b600a2050 con 0x7f9b44002840 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b44006590 con 0x7f9b44002840 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9b50002350 con 0x7f9b44002840 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 monclient: dump: 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.313 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 138873806 0 0) 0x7f9b44006590 con 0x7f9b44002840 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b4400a140 con 0x7f9b44002840 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.292+0000 7f9b57fff700 10 client.74559.objecter ms_handle_connect 0x7f9b40038610 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.296+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f9b50005810 con 0x7f9b44002840 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.296+0000 7f9b57fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.296+0000 7f9b57fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.296+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6789/0 conn(0x7f9b44002840 legacy=0x7f9b44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.296+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6789/0 conn(0x7f9b44002840 legacy=0x7f9b44004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.304+0000 7f9b64aa3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:48:07.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.304+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9b44009f10 con 0x7f9b44006900 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 client.74559.objecter ms_handle_connect 0x7f9b44006900 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b5f7fe700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6789/0 conn(0x7f9b44006900 legacy=0x7f9b44006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 client.74559.objecter resend_mon_ops 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239989926 0 0) 0x7f9b44009f10 con 0x7f9b44006900 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 monclient(hunting): my global_id is 74565 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b44006590 con 0x7f9b44006900 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3303578417 0 0) 0x7f9b44006590 con 0x7f9b44006900 2024-09-18T11:48:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b44009f10 con 0x7f9b44006900 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9b500029b0 con 0x7f9b44006900 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 monclient: dump: 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.317 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4074790388 0 0) 0x7f9b44009f10 con 0x7f9b44006900 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9b44008760 con 0x7f9b44006900 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4003 (unknown 0 0 578262818) 0x7f9b50002c80 con 0x7f9b44006900 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6789/0 conn(0x7f9b44006900 legacy=0x7f9b44006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b57fff700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6789/0 conn(0x7f9b44006900 legacy=0x7f9b44006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:48:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 20, 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:48:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:48:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:48:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:48:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:48:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:48:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:48:07.330 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:48:07.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b64aa3700 10 librados: watch_flush enter 2024-09-18T11:48:07.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b64aa3700 10 librados: watch_flush exit 2024-09-18T11:48:07.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b64aa3700 20 client.74559.objecter shutdown clearing up homeless session... 2024-09-18T11:48:07.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b64aa3700 10 client.74559.objecter successfully canceled tick 2024-09-18T11:48:07.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.129:6812/20806 conn(0x7f9b40038610 legacy=0x7f9b4003aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b64aa3700 10 monclient: shutdown 2024-09-18T11:48:07.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b5ffff700 1 -- 172.21.15.112:0/47627074 reap_dead start 2024-09-18T11:48:07.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b64aa3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:07.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.308+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 >> v1:172.21.15.112:6789/0 conn(0x7f9b600a0e60 legacy=0x7f9b60149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.312+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 shutdown_connections 2024-09-18T11:48:07.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.312+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 >> 172.21.15.112:0/47627074 conn(0x7f9b60068100 msgr2=0x7f9b600a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:07.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.312+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 shutdown_connections 2024-09-18T11:48:07.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.312+0000 7f9b64aa3700 1 -- 172.21.15.112:0/47627074 wait complete. 2024-09-18T11:48:07.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.312+0000 7f9b64aa3700 1 librados: shutdown 2024-09-18T11:48:07.333 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:48:07.333 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:48:07.334 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:48:07.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient: get_monmap_and_config 2024-09-18T11:48:07.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient: build_initial_monmap 2024-09-18T11:48:07.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient: monmap: 2024-09-18T11:48:07.445 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:07.445 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.445 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:07.444896+0000 2024-09-18T11:48:07.445 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:07.444896+0000 2024-09-18T11:48:07.446 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:07.446 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.446 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:07.446 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:07.446 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:07.446 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 1 Processor -- start 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 1 -- start start 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient: init 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient(hunting): picked mon.noname-a con 0x7ff2b80a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient(hunting): picked mon.noname-c con 0x7ff2b8096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient(hunting): picked mon.noname-b con 0x7ff2b813ca20 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b806e910 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b806e630 con 0x7ff2b8096a60 2024-09-18T11:48:07.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b80a2270 con 0x7ff2b813ca20 2024-09-18T11:48:07.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:07.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bdfd4700 10 monclient(hunting): authenticate will time out at 1059.286303s 2024-09-18T11:48:07.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bcfd2700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff2b80a1ea0 0x7ff2b806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34142/0 (socket says 172.21.15.112:34142) 2024-09-18T11:48:07.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.440+0000 7ff2bcfd2700 1 -- 172.21.15.112:0/2937062441 learned_addr learned my addr 172.21.15.112:0/2937062441 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bcfd2700 1 -- 172.21.15.112:0/2937062441 >> v1:172.21.15.112:6789/0 conn(0x7ff2b80a1ea0 legacy=0x7ff2b806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b7fff700 1 -- 172.21.15.112:0/2937062441 >> v1:172.21.15.129:6789/0 conn(0x7ff2b8096a60 legacy=0x7ff2b806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bd7d3700 1 -- 172.21.15.112:0/2937062441 >> v1:172.21.15.112:6790/0 conn(0x7ff2b813ca20 legacy=0x7ff2b813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335203956 0 0) 0x7ff2b806e910 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient(hunting): my global_id is 74568 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a0003570 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030271574 0 0) 0x7ff2a0003570 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 >> v1:172.21.15.112:6790/0 conn(0x7ff2b813ca20 legacy=0x7ff2b813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 >> v1:172.21.15.129:6789/0 conn(0x7ff2b8096a60 legacy=0x7ff2b806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: _finish_hunting 0 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 monclient: found mon.noname-a 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b806eed0 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: _finish_auth 0 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff2ac001a00 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: dump: 2024-09-18T11:48:07.452 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 5 monclient: authenticate success, global_id 74568 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:07.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2ac001c60 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 1 -- 172.21.15.112:0/2937062441 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff2ac004a90 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 4 set_mon_vals no callback set 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2b77fe700 10 monclient: dump: 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.454 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.455 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.455 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.455 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.455 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 10 monclient: shutdown 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/2937062441 >> v1:172.21.15.112:6789/0 conn(0x7ff2b80a1ea0 legacy=0x7ff2b806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/2937062441 shutdown_connections 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/2937062441 >> 172.21.15.112:0/2937062441 conn(0x7ff2b8067da0 msgr2=0x7ff2b80a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/2937062441 shutdown_connections 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.444+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/2937062441 wait complete. 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient: build_initial_monmap 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient: monmap: 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:07.461 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 librados: starting msgr at 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 librados: starting objecter 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 Processor -- start 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 -- start start 2024-09-18T11:48:07.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 librados: setting wanted keys 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 librados: calling monclient init 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient: init 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient(hunting): picked mon.noname-b con 0x7ff2b8096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient(hunting): picked mon.noname-a con 0x7ff2b80a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient(hunting): picked mon.noname-c con 0x7ff2b813ca20 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b806e110 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b809dde0 con 0x7ff2b8096a60 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b8126cb0 con 0x7ff2b813ca20 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bdfd4700 10 monclient(hunting): authenticate will time out at 1059.294149s 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bd7d3700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff2b813ca20 0x7ff2b814ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33038/0 (socket says 172.21.15.112:33038) 2024-09-18T11:48:07.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2b7fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff2b80a1ea0 0x7ff2b8123990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34158/0 (socket says 172.21.15.112:34158) 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.448+0000 7ff2bd7d3700 1 -- 172.21.15.112:0/1654188204 learned_addr learned my addr 172.21.15.112:0/1654188204 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b7fff700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6789/0 conn(0x7ff2b80a1ea0 legacy=0x7ff2b8123990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 client.?.objecter ms_handle_connect 0x7ff2b80a1ea0 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 client.?.objecter ms_handle_connect 0x7ff2b813ca20 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bd7d3700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6790/0 conn(0x7ff2b813ca20 legacy=0x7ff2b814ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bcfd2700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.129:6789/0 conn(0x7ff2b8096a60 legacy=0x7ff2b8123280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 client.?.objecter ms_handle_connect 0x7ff2b8096a60 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124708714 0 0) 0x7ff2b806e110 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): my global_id is 74571 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a4004f10 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684288327 0 0) 0x7ff2b8126cb0 con 0x7ff2b813ca20 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): my global_id is 84143 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b806e110 con 0x7ff2b813ca20 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793996749 0 0) 0x7ff2b809dde0 con 0x7ff2b8096a60 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): my global_id is 17866 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b8126cb0 con 0x7ff2b8096a60 2024-09-18T11:48:07.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4075363684 0 0) 0x7ff2b806e110 con 0x7ff2b813ca20 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2b809dde0 con 0x7ff2b813ca20 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff2b0001bc0 con 0x7ff2b813ca20 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): dump: 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.466 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2437332004 0 0) 0x7ff2a4004f10 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2b806e110 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff2a8001f80 con 0x7ff2b80a1ea0 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient(hunting): dump: 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.467 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1627290172 0 0) 0x7ff2b809dde0 con 0x7ff2b813ca20 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.129:6789/0 conn(0x7ff2b8096a60 legacy=0x7ff2b8123280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6789/0 conn(0x7ff2b80a1ea0 legacy=0x7ff2b8123990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 monclient: found mon.c 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:07.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b813c7e0 con 0x7ff2b813ca20 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient: _finish_auth 0 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 5 monclient: authenticate success, global_id 84143 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 10 monclient: _renew_subs 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b8124c30 con 0x7ff2b813ca20 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 10 client.84143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 10 monclient: _renew_subs 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b81259a0 con 0x7ff2b813ca20 2024-09-18T11:48:07.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 1 librados: init done 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2b0002230 con 0x7ff2b813ca20 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff2b0002540 con 0x7ff2b813ca20 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b6ffd700 4 set_mon_vals no callback set 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 monclient: dump: 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.470 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ff2b00026a0 con 0x7ff2b813ca20 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff298006590 con 0x7ff298002840 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(166..166 src has 1..166) v4 ==== 6835+0+0 (unknown 3653208750 0 0) 0x7ff2b00443b0 con 0x7ff2b813ca20 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 10 client.84143.objecter ms_dispatch 0x7ff2b8000fd0 osd_map(166..166 src has 1..166) v4 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 3 client.84143.objecter handle_osd_map got epochs [166,166] > 0 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 3 client.84143.objecter handle_osd_map decoding full epoch 166 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b4ff9700 20 client.84143.objecter dump_active .. 0 homeless 2024-09-18T11:48:07.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.452+0000 7ff2b7fff700 10 monclient: get_auth_request con 0x7ff2a4038440 auth_method 0 2024-09-18T11:48:07.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2bcfd2700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6790/0 conn(0x7ff298002840 legacy=0x7ff298004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 client.84143.objecter ms_handle_connect 0x7ff298002840 2024-09-18T11:48:07.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 client.84143.objecter resend_mon_ops 2024-09-18T11:48:07.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721221670 0 0) 0x7ff298006590 con 0x7ff298002840 2024-09-18T11:48:07.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 monclient(hunting): my global_id is 84146 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b809dde0 con 0x7ff298002840 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1121845907 0 0) 0x7ff2b809dde0 con 0x7ff298002840 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff298006590 con 0x7ff298002840 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff2ac001740 con 0x7ff298002840 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 monclient: dump: 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.474 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 client.84143.objecter ms_handle_connect 0x7ff2a4038440 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3220999527 0 0) 0x7ff298006590 con 0x7ff298002840 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff29800a140 con 0x7ff298002840 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7ff2ac001920 con 0x7ff298002840 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6790/0 conn(0x7ff298002840 legacy=0x7ff298004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.456+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6790/0 conn(0x7ff298002840 legacy=0x7ff298004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:07.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2bdfd4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:48:07.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff298007550 con 0x7ff298006900 2024-09-18T11:48:07.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2bcfd2700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6790/0 conn(0x7ff298006900 legacy=0x7ff298006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 10 client.84143.objecter ms_handle_connect 0x7ff298006900 2024-09-18T11:48:07.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 10 client.84143.objecter resend_mon_ops 2024-09-18T11:48:07.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171658800 0 0) 0x7ff298007550 con 0x7ff298006900 2024-09-18T11:48:07.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 10 monclient(hunting): my global_id is 84149 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff298006590 con 0x7ff298006900 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 292653020 0 0) 0x7ff298006590 con 0x7ff298006900 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff298007550 con 0x7ff298006900 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff2ac005390 con 0x7ff298006900 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.468+0000 7ff2b4ff9700 10 monclient: dump: 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.478 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 348335110 0 0) 0x7ff298007550 con 0x7ff298006900 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff2980087b0 con 0x7ff298006900 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 2328392999) 0x7ff2ac006f00 con 0x7ff298006900 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2b4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2b4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6790/0 conn(0x7ff298006900 legacy=0x7ff298006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2b4ff9700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6790/0 conn(0x7ff298006900 legacy=0x7ff298006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 20, 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.480 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.481 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.482 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:48:07.483 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.484 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:48:07.485 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:07.486 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.487 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.489 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:48:07.490 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:48:07.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 10 librados: watch_flush enter 2024-09-18T11:48:07.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 10 librados: watch_flush exit 2024-09-18T11:48:07.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 20 client.84143.objecter shutdown clearing up homeless session... 2024-09-18T11:48:07.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 10 client.84143.objecter successfully canceled tick 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.129:6812/20806 conn(0x7ff2a4038440 legacy=0x7ff2a403a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 10 monclient: shutdown 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 >> v1:172.21.15.112:6790/0 conn(0x7ff2b813ca20 legacy=0x7ff2b814ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bd7d3700 1 -- 172.21.15.112:0/1654188204 reap_dead start 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bd7d3700 1 -- 172.21.15.112:0/1654188204 reap_dead start 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 shutdown_connections 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 >> 172.21.15.112:0/1654188204 conn(0x7ff2b8067da0 msgr2=0x7ff2b809fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 shutdown_connections 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 1 -- 172.21.15.112:0/1654188204 wait complete. 2024-09-18T11:48:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.472+0000 7ff2bdfd4700 1 librados: shutdown 2024-09-18T11:48:07.494 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:48:07.494 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:48:07.494 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:48:07.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient: get_monmap_and_config 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient: build_initial_monmap 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient: monmap: 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:07.605553+0000 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:07.605553+0000 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:07.606 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:07.607 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 1 Processor -- start 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 1 -- start start 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient: init 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f54700a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f5470096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f547013ca20 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547006e910 con 0x7f547013ca20 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547006e630 con 0x7f5470096a60 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54700a2270 con 0x7f54700a1ea0 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546ffff700 10 monclient(hunting): authenticate will time out at 1059.446484s 2024-09-18T11:48:07.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546f7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f547013ca20 0x7f547013ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34164/0 (socket says 172.21.15.112:34164) 2024-09-18T11:48:07.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.600+0000 7f546f7fe700 1 -- 172.21.15.112:0/3597092247 learned_addr learned my addr 172.21.15.112:0/3597092247 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546f7fe700 1 -- 172.21.15.112:0/3597092247 >> v1:172.21.15.112:6789/0 conn(0x7f547013ca20 legacy=0x7f547013ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546effd700 1 -- 172.21.15.112:0/3597092247 >> v1:172.21.15.112:6790/0 conn(0x7f54700a1ea0 legacy=0x7f547006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203437741 0 0) 0x7f547006e910 con 0x7f547013ca20 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): my global_id is 74574 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5458003570 con 0x7f547013ca20 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546e7fc700 1 -- 172.21.15.112:0/3597092247 >> v1:172.21.15.129:6789/0 conn(0x7f5470096a60 legacy=0x7f547006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155049053 0 0) 0x7f54700a2270 con 0x7f54700a1ea0 2024-09-18T11:48:07.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): my global_id is 84152 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f547006e910 con 0x7f54700a1ea0 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610310208 0 0) 0x7f547006e630 con 0x7f5470096a60 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient(hunting): my global_id is 17869 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54700a2270 con 0x7f5470096a60 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1172718029 0 0) 0x7f5458003570 con 0x7f547013ca20 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 >> v1:172.21.15.112:6790/0 conn(0x7f54700a1ea0 legacy=0x7f547006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 >> v1:172.21.15.129:6789/0 conn(0x7f5470096a60 legacy=0x7f547006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: _finish_hunting 0 2024-09-18T11:48:07.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 monclient: found mon.noname-a 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547006eed0 con 0x7f547013ca20 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: _finish_auth 0 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 5 monclient: authenticate success, global_id 74574 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5460001940 con 0x7f547013ca20 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: dump: 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.615 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5460001bd0 con 0x7f547013ca20 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 4 set_mon_vals no callback set 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 1 -- 172.21.15.112:0/3597092247 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5460004a90 con 0x7f547013ca20 2024-09-18T11:48:07.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546dffb700 10 monclient: dump: 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 10 monclient: shutdown 2024-09-18T11:48:07.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 -- 172.21.15.112:0/3597092247 >> v1:172.21.15.112:6789/0 conn(0x7f547013ca20 legacy=0x7f547013ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 -- 172.21.15.112:0/3597092247 shutdown_connections 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 -- 172.21.15.112:0/3597092247 >> 172.21.15.112:0/3597092247 conn(0x7f5470067da0 msgr2=0x7f54700a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 -- 172.21.15.112:0/3597092247 shutdown_connections 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 -- 172.21.15.112:0/3597092247 wait complete. 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 10 monclient: build_initial_monmap 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 10 monclient: monmap: 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:07.618 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 librados: starting msgr at 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 librados: starting objecter 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 Processor -- start 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 -- start start 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 librados: setting wanted keys 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 1 librados: calling monclient init 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.604+0000 7f546ffff700 10 monclient: init 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:48:07.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f5470096a60 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f54700a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f547013ca20 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5470125070 con 0x7f5470096a60 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5470125760 con 0x7f54700a1ea0 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547006e110 con 0x7f547013ca20 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 10 monclient(hunting): authenticate will time out at 1059.452360s 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546f7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f547013ca20 0x7f5470123d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33056/0 (socket says 172.21.15.112:33056) 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546effd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5470096a60 0x7f5470122f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34178/0 (socket says 172.21.15.112:34178) 2024-09-18T11:48:07.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546f7fe700 1 -- 172.21.15.112:0/509532927 learned_addr learned my addr 172.21.15.112:0/509532927 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:07.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546f7fe700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.112:6790/0 conn(0x7f547013ca20 legacy=0x7f5470123d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 client.?.objecter ms_handle_connect 0x7f547013ca20 2024-09-18T11:48:07.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 client.?.objecter ms_handle_connect 0x7f5470096a60 2024-09-18T11:48:07.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546effd700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.112:6789/0 conn(0x7f5470096a60 legacy=0x7f5470122f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 client.?.objecter ms_handle_connect 0x7f54700a1ea0 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546e7fc700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6789/0 conn(0x7f54700a1ea0 legacy=0x7f5470123670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533439612 0 0) 0x7f5470125070 con 0x7f5470096a60 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): my global_id is 74577 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f545c004f10 con 0x7f5470096a60 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463340191 0 0) 0x7f547006e110 con 0x7f547013ca20 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): my global_id is 84155 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5470125070 con 0x7f547013ca20 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815045332 0 0) 0x7f5470125760 con 0x7f54700a1ea0 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): my global_id is 17872 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f547006e110 con 0x7f54700a1ea0 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 6809523 0 0) 0x7f545c004f10 con 0x7f5470096a60 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5470125760 con 0x7f5470096a60 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5464001f00 con 0x7f5470096a60 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): dump: 2024-09-18T11:48:07.627 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3810367095 0 0) 0x7f5470125070 con 0x7f547013ca20 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f545c004f10 con 0x7f547013ca20 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5460004830 con 0x7f547013ca20 2024-09-18T11:48:07.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient(hunting): dump: 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1299164445 0 0) 0x7f5470125760 con 0x7f5470096a60 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.112:6790/0 conn(0x7f547013ca20 legacy=0x7f5470123d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6789/0 conn(0x7f54700a1ea0 legacy=0x7f5470123670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 monclient: found mon.a 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547009dbd0 con 0x7f5470096a60 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 10 monclient: _finish_auth 0 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f5456ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.608+0000 7f546ffff700 5 monclient: authenticate success, global_id 74577 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 10 monclient: _renew_subs 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5470124af0 con 0x7f5470096a60 2024-09-18T11:48:07.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 10 client.74577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 10 monclient: _renew_subs 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f547013c720 con 0x7f5470096a60 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 1 librados: init done 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5464002390 con 0x7f5470096a60 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f54640026e0 con 0x7f5470096a60 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546d7fa700 4 set_mon_vals no callback set 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 10 monclient: dump: 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.631 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5450006590 con 0x7f5450002840 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5464002940 con 0x7f5470096a60 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(166..166 src has 1..166) v4 ==== 6835+0+0 (unknown 3653208750 0 0) 0x7f54640443f0 con 0x7f5470096a60 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 10 client.74577.objecter ms_dispatch 0x7f5470000fd0 osd_map(166..166 src has 1..166) v4 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 3 client.74577.objecter handle_osd_map got epochs [166,166] > 0 2024-09-18T11:48:07.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 3 client.74577.objecter handle_osd_map decoding full epoch 166 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 20 client.74577.objecter dump_active .. 0 homeless 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546f7fe700 10 monclient: get_auth_request con 0x7f545c0384b0 auth_method 0 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 10 client.74577.objecter ms_handle_connect 0x7f5450002840 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f546e7fc700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6789/0 conn(0x7f5450002840 legacy=0x7f5450004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.612+0000 7f5456ffd700 10 client.74577.objecter resend_mon_ops 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786445970 0 0) 0x7f5450006590 con 0x7f5450002840 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 monclient(hunting): my global_id is 17875 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5470125760 con 0x7f5450002840 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1271305633 0 0) 0x7f5470125760 con 0x7f5450002840 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5450006590 con 0x7f5450002840 2024-09-18T11:48:07.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5468001ec0 con 0x7f5450002840 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 monclient: dump: 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 client.74577.objecter ms_handle_connect 0x7f545c0384b0 2024-09-18T11:48:07.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 734614985 0 0) 0x7f5450006590 con 0x7f5450002840 2024-09-18T11:48:07.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f545000a140 con 0x7f5450002840 2024-09-18T11:48:07.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5468005b90 con 0x7f5450002840 2024-09-18T11:48:07.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:07.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6789/0 conn(0x7f5450002840 legacy=0x7f5450004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.616+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6789/0 conn(0x7f5450002840 legacy=0x7f5450004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:07.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.628+0000 7f546ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:48:07.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.628+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5450007550 con 0x7f5450006900 2024-09-18T11:48:07.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546e7fc700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6789/0 conn(0x7f5450006900 legacy=0x7f5450006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 client.74577.objecter ms_handle_connect 0x7f5450006900 2024-09-18T11:48:07.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 client.74577.objecter resend_mon_ops 2024-09-18T11:48:07.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750134919 0 0) 0x7f5450007550 con 0x7f5450006900 2024-09-18T11:48:07.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 monclient(hunting): my global_id is 17878 2024-09-18T11:48:07.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450006590 con 0x7f5450006900 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3665691666 0 0) 0x7f5450006590 con 0x7f5450006900 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5450007550 con 0x7f5450006900 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5468002790 con 0x7f5450006900 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 monclient: dump: 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.639 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.640 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.640 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.640 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.640 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3187874804 0 0) 0x7f5450007550 con 0x7f5450006900 2024-09-18T11:48:07.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f54500087b0 con 0x7f5450006900 2024-09-18T11:48:07.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 246798294) 0x7f54680029b0 con 0x7f5450006900 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6789/0 conn(0x7f5450006900 legacy=0x7f5450006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f5456ffd700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6789/0 conn(0x7f5450006900 legacy=0x7f5450006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 20, 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:48:07.641 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.642 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.643 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:48:07.644 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:07.645 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:07.646 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.647 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:07.649 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:48:07.650 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.651 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 6 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:07.652 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:07.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:07.653 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:48:07.653 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:48:07.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 10 librados: watch_flush enter 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 10 librados: watch_flush exit 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 20 client.74577.objecter shutdown clearing up homeless session... 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 10 client.74577.objecter successfully canceled tick 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.129:6812/20806 conn(0x7f545c0384b0 legacy=0x7f545c03a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 10 monclient: shutdown 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 >> v1:172.21.15.112:6789/0 conn(0x7f5470096a60 legacy=0x7f5470122f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546f7fe700 1 -- 172.21.15.112:0/509532927 reap_dead start 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546f7fe700 1 -- 172.21.15.112:0/509532927 reap_dead start 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 shutdown_connections 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.632+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 >> 172.21.15.112:0/509532927 conn(0x7f5470067da0 msgr2=0x7f547009a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:07.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.636+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 shutdown_connections 2024-09-18T11:48:07.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.636+0000 7f546ffff700 1 -- 172.21.15.112:0/509532927 wait complete. 2024-09-18T11:48:07.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.636+0000 7f546ffff700 1 librados: shutdown 2024-09-18T11:48:07.656 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:48:07.656 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T11:48:07.656 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T11:48:07.656 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T11:48:07.656 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T11:48:07.656 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T11:48:07.657 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:48:07.669 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T11:48:07.670 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T11:48:07.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:48:07.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.760+0000 7f068261e700 10 monclient: get_monmap_and_config 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.760+0000 7f068261e700 10 monclient: build_initial_monmap 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.760+0000 7f068261e700 10 monclient: monmap: 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:07.766776+0000 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:07.766776+0000 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:07.768 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.760+0000 7f068261e700 1 Processor -- start 2024-09-18T11:48:07.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.760+0000 7f068261e700 1 -- start start 2024-09-18T11:48:07.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.760+0000 7f068261e700 10 monclient: init 2024-09-18T11:48:07.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:07.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:48:07.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 10 monclient(hunting): picked mon.noname-b con 0x7f067c06aa40 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:07.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 10 monclient(hunting): picked mon.noname-a con 0x7f067c0a1e80 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:07.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 10 monclient(hunting): picked mon.noname-c con 0x7f067c1386e0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:07.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c097270 con 0x7f067c0a1e80 2024-09-18T11:48:07.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c06ae10 con 0x7f067c1386e0 2024-09-18T11:48:07.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c097550 con 0x7f067c06aa40 2024-09-18T11:48:07.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:07.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 10 monclient(hunting): authenticate will time out at 1059.608759s 2024-09-18T11:48:07.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f0673fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f067c0a1e80 legacy=0x7f067c0a1160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:07.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067bfff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f067c06aa40 0x7f067c0a0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33086/0 (socket says 172.21.15.112:33086) 2024-09-18T11:48:07.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067bfff700 1 -- 172.21.15.112:0/2165188119 learned_addr learned my addr 172.21.15.112:0/2165188119 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067bfff700 1 -- 172.21.15.112:0/2165188119 >> v1:172.21.15.112:6790/0 conn(0x7f067c06aa40 legacy=0x7f067c0a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156852756 0 0) 0x7f067c097550 con 0x7f067c06aa40 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient(hunting): my global_id is 84158 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f0680bbb700 1 -- 172.21.15.112:0/2165188119 >> v1:172.21.15.129:6789/0 conn(0x7f067c1386e0 legacy=0x7f067c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0668003570 con 0x7f067c06aa40 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2791797204 0 0) 0x7f0668003570 con 0x7f067c06aa40 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 >> v1:172.21.15.129:6789/0 conn(0x7f067c1386e0 legacy=0x7f067c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 >> v1:172.21.15.112:6789/0 conn(0x7f067c0a1e80 legacy=0x7f067c0a1160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:48:07.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 1 monclient: found mon.noname-b 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c096a60 con 0x7f067c06aa40 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient: _finish_auth 0 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f066c001a00 con 0x7f067c06aa40 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f067b7fe700 10 monclient: dump: 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.780 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 5 monclient: authenticate success, global_id 84158 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.764+0000 7f068261e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f066c001be0 con 0x7f067c06aa40 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f067b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:07.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 4 set_mon_vals no callback set 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f067b7fe700 1 -- 172.21.15.112:0/2165188119 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f066c004a90 con 0x7f067c06aa40 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f067b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f067b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f067b7fe700 10 monclient: dump: 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.782 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient: shutdown 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- 172.21.15.112:0/2165188119 >> v1:172.21.15.112:6790/0 conn(0x7f067c06aa40 legacy=0x7f067c0a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- 172.21.15.112:0/2165188119 shutdown_connections 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- 172.21.15.112:0/2165188119 >> 172.21.15.112:0/2165188119 conn(0x7f067c0a5d10 msgr2=0x7f067c0a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- 172.21.15.112:0/2165188119 shutdown_connections 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- 172.21.15.112:0/2165188119 wait complete. 2024-09-18T11:48:07.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient: build_initial_monmap 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient: monmap: 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 librados: starting msgr at 2024-09-18T11:48:07.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 librados: starting objecter 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 Processor -- start 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- start start 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 librados: setting wanted keys 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 librados: calling monclient init 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient: init 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient(hunting): picked mon.noname-b con 0x7f067c06aa40 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient(hunting): picked mon.noname-a con 0x7f067c0a1e80 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient(hunting): picked mon.noname-c con 0x7f067c1386e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c09b500 con 0x7f067c0a1e80 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c09bac0 con 0x7f067c06aa40 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c09a330 con 0x7f067c1386e0 2024-09-18T11:48:07.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f068261e700 10 monclient(hunting): authenticate will time out at 1059.615103s 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f0673fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f067c0a1e80 legacy=0x7f067c1468e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f0680bbb700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f067c1386e0 0x7f067c14a010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:33090/0 (socket says 172.21.15.112:33090) 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.768+0000 7f0680bbb700 1 -- 172.21.15.112:0/1581022597 learned_addr learned my addr 172.21.15.112:0/1581022597 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 client.?.objecter ms_handle_connect 0x7f067c1386e0 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0680bbb700 1 -- 172.21.15.112:0/1581022597 >> v1:172.21.15.112:6790/0 conn(0x7f067c1386e0 legacy=0x7f067c14a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f067bfff700 1 -- 172.21.15.112:0/1581022597 >> v1:172.21.15.129:6789/0 conn(0x7f067c06aa40 legacy=0x7f067c09dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:07.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 DispatchQueue::entry inject delay of 0.002000 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 client.?.objecter ms_handle_connect 0x7f067c06aa40 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463228374 0 0) 0x7f067c09a330 con 0x7f067c1386e0 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 monclient(hunting): my global_id is 84161 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f065c004f10 con 0x7f067c1386e0 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154391543 0 0) 0x7f067c09bac0 con 0x7f067c06aa40 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 10 monclient(hunting): my global_id is 17884 2024-09-18T11:48:07.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.772+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067c09a330 con 0x7f067c06aa40 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 544496300 0 0) 0x7f065c004f10 con 0x7f067c1386e0 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f067c09bac0 con 0x7f067c1386e0 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0674001f80 con 0x7f067c1386e0 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient(hunting): dump: 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1724211740 0 0) 0x7f067c09a330 con 0x7f067c06aa40 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f065c004f10 con 0x7f067c06aa40 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f066c0017a0 con 0x7f067c06aa40 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0680bbb700 1 --1- 172.21.15.112:0/1581022597 >> v1:172.21.15.112:6790/0 conn(0x7f067c1386e0 0x7f067c14a010 :-1 s=OPENED pgs=8821 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f067c09bac0 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient(hunting): dump: 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.792 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2968822083 0 0) 0x7f065c004f10 con 0x7f067c06aa40 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 >> v1:172.21.15.112:6790/0 conn(0x7f067c1386e0 legacy=0x7f067c14a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 >> v1:172.21.15.112:6789/0 conn(0x7f067c0a1e80 legacy=0x7f067c1468e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:48:07.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 monclient: found mon.b 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c09a4c0 con 0x7f067c06aa40 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient: _finish_auth 0 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 5 monclient: authenticate success, global_id 17884 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2053534125 0 0) 0x7f067c09bac0 con 0x7f067c1386e0 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 10 monclient: _renew_subs 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c09c490 con 0x7f067c06aa40 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 10 client.17884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 10 monclient: _renew_subs 2024-09-18T11:48:07.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c098d20 con 0x7f067c06aa40 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 1 librados: init done 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f066c0019b0 con 0x7f067c06aa40 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f066c005160 con 0x7f067c06aa40 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f067a7fc700 4 set_mon_vals no callback set 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 monclient: dump: 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:07.795 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06600050b0 con 0x7f067c06aa40 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f066c00f9e0 con 0x7f067c06aa40 2024-09-18T11:48:07.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(166..166 src has 1..166) v4 ==== 6835+0+0 (unknown 3653208750 0 0) 0x7f066c044500 con 0x7f067c06aa40 2024-09-18T11:48:07.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 10 client.17884.objecter ms_dispatch 0x7f067c000fd0 osd_map(166..166 src has 1..166) v4 2024-09-18T11:48:07.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 3 client.17884.objecter handle_osd_map got epochs [166,166] > 0 2024-09-18T11:48:07.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 3 client.17884.objecter handle_osd_map decoding full epoch 166 2024-09-18T11:48:07.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.776+0000 7f0678ff9700 20 client.17884.objecter dump_active .. 0 homeless 2024-09-18T11:48:07.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.780+0000 7f0673fff700 10 monclient: get_auth_request con 0x7f065c038330 auth_method 0 2024-09-18T11:48:07.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.780+0000 7f0678ff9700 10 client.17884.objecter ms_handle_connect 0x7f065c038330 2024-09-18T11:48:07.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.792+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f066c014bf0 con 0x7f067c06aa40 2024-09-18T11:48:07.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.792+0000 7f0678ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:07.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:07.792+0000 7f0678ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:08.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f068261e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:48:08.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f068261e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:08.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f068261e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:08.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f06600058e0 con 0x7f067c06aa40 2024-09-18T11:48:08.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f0678ff9700 1 -- 172.21.15.112:0/1581022597 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3431562589) 0x7f066c0145f0 con 0x7f067c06aa40 2024-09-18T11:48:08.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f0678ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:08.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f0678ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:08.026 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f068261e700 10 librados: watch_flush enter 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f068261e700 10 librados: watch_flush exit 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f068261e700 20 client.17884.objecter shutdown clearing up homeless session... 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.020+0000 7f068261e700 10 client.17884.objecter successfully canceled tick 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 >> v1:172.21.15.129:6812/20806 conn(0x7f065c038330 legacy=0x7f065c03a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 10 monclient: shutdown 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 >> v1:172.21.15.129:6789/0 conn(0x7f067c06aa40 legacy=0x7f067c09dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:08.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 shutdown_connections 2024-09-18T11:48:08.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 >> 172.21.15.112:0/1581022597 conn(0x7f067c0a5d10 msgr2=0x7f067c13ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:08.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 shutdown_connections 2024-09-18T11:48:08.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 1 -- 172.21.15.112:0/1581022597 wait complete. 2024-09-18T11:48:08.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:08.024+0000 7f068261e700 1 librados: shutdown 2024-09-18T11:48:11.042 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:48:11.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.144+0000 7fb1191c0700 10 monclient: get_monmap_and_config 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.144+0000 7fb1191c0700 10 monclient: build_initial_monmap 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.144+0000 7fb1191c0700 10 monclient: monmap: 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:11.151423+0000 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:11.151423+0000 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:11.152 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:11.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.144+0000 7fb1191c0700 1 Processor -- start 2024-09-18T11:48:11.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.144+0000 7fb1191c0700 1 -- start start 2024-09-18T11:48:11.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.144+0000 7fb1191c0700 10 monclient: init 2024-09-18T11:48:11.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:11.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:48:11.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 10 monclient(hunting): picked mon.noname-b con 0x7fb1140a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:11.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 10 monclient(hunting): picked mon.noname-a con 0x7fb11412edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:11.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 10 monclient(hunting): picked mon.noname-c con 0x7fb11412f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140981a0 con 0x7fb11412edc0 2024-09-18T11:48:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11406c120 con 0x7fb11412f8a0 2024-09-18T11:48:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11406eef0 con 0x7fb1140a2040 2024-09-18T11:48:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 10 monclient(hunting): authenticate will time out at 1062.992311s 2024-09-18T11:48:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1137fe700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fb11412edc0 legacy=0x7fb11412f190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb113fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb1140a2040 0x7fb114097a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38532/0 (socket says 172.21.15.112:38532) 2024-09-18T11:48:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb113fff700 1 -- 172.21.15.112:0/817970503 learned_addr learned my addr 172.21.15.112:0/817970503 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:11.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb113fff700 1 -- 172.21.15.112:0/817970503 >> v1:172.21.15.112:6790/0 conn(0x7fb1140a2040 legacy=0x7fb114097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337473006 0 0) 0x7fb11406eef0 con 0x7fb1140a2040 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient(hunting): my global_id is 84164 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb100003570 con 0x7fb1140a2040 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1189bf700 1 -- 172.21.15.112:0/817970503 >> v1:172.21.15.129:6789/0 conn(0x7fb11412f8a0 legacy=0x7fb114137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 968666463 0 0) 0x7fb100003570 con 0x7fb1140a2040 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 >> v1:172.21.15.129:6789/0 conn(0x7fb11412f8a0 legacy=0x7fb114137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 >> v1:172.21.15.112:6789/0 conn(0x7fb11412edc0 legacy=0x7fb11412f190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:48:11.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 monclient: found mon.noname-b 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb114096dd0 con 0x7fb1140a2040 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: _finish_auth 0 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb10c001d80 con 0x7fb1140a2040 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 5 monclient: authenticate success, global_id 84164 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:48:11.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: dump: 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb10c001f60 con 0x7fb1140a2040 2024-09-18T11:48:11.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb1191c0700 4 set_mon_vals no callback set 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 1 -- 172.21.15.112:0/817970503 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb10c004e10 con 0x7fb1140a2040 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.148+0000 7fb112ffd700 10 monclient: dump: 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:11.161 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient: shutdown 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- 172.21.15.112:0/817970503 >> v1:172.21.15.112:6790/0 conn(0x7fb1140a2040 legacy=0x7fb114097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- 172.21.15.112:0/817970503 shutdown_connections 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- 172.21.15.112:0/817970503 >> 172.21.15.112:0/817970503 conn(0x7fb114067da0 msgr2=0x7fb1140a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- 172.21.15.112:0/817970503 shutdown_connections 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- 172.21.15.112:0/817970503 wait complete. 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient: build_initial_monmap 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient: monmap: 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:11.162 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 librados: starting msgr at 2024-09-18T11:48:11.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 librados: starting objecter 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 Processor -- start 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- start start 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 librados: setting wanted keys 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 librados: calling monclient init 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient: init 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient(hunting): picked mon.noname-a con 0x7fb1140a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient(hunting): picked mon.noname-c con 0x7fb11412edc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient(hunting): picked mon.noname-b con 0x7fb11412f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:11.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb114119ce0 con 0x7fb1140a2040 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11406f110 con 0x7fb11412f8a0 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb113fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fb1140a2040 legacy=0x7fb114141290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140a0500 con 0x7fb11412edc0 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1191c0700 10 monclient(hunting): authenticate will time out at 1062.998900s 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1137fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb11412edc0 0x7fb11412e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38534/0 (socket says 172.21.15.112:38534) 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.152+0000 7fb1137fe700 1 -- 172.21.15.112:0/2526394076 learned_addr learned my addr 172.21.15.112:0/2526394076 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1137fe700 1 -- 172.21.15.112:0/2526394076 >> v1:172.21.15.112:6790/0 conn(0x7fb11412edc0 legacy=0x7fb11412e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 client.?.objecter ms_handle_connect 0x7fb11412edc0 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305620131 0 0) 0x7fb1140a0500 con 0x7fb11412edc0 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:11.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): my global_id is 84167 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4004f10 con 0x7fb11412edc0 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 client.?.objecter ms_handle_connect 0x7fb11412f8a0 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1189bf700 1 -- 172.21.15.112:0/2526394076 >> v1:172.21.15.129:6789/0 conn(0x7fb11412f8a0 legacy=0x7fb114149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478243683 0 0) 0x7fb11406f110 con 0x7fb11412f8a0 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): my global_id is 17890 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1140a0500 con 0x7fb11412f8a0 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3126068746 0 0) 0x7fb0f4004f10 con 0x7fb11412edc0 2024-09-18T11:48:11.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb11406f110 con 0x7fb11412edc0 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb0fc0045a0 con 0x7fb11412edc0 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): dump: 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:11.171 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3361025282 0 0) 0x7fb11406f110 con 0x7fb11412edc0 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 >> v1:172.21.15.129:6789/0 conn(0x7fb11412f8a0 legacy=0x7fb114149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 >> v1:172.21.15.112:6789/0 conn(0x7fb1140a2040 legacy=0x7fb114141290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient: _finish_hunting 0 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 monclient: found mon.c 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11411a330 con 0x7fb11412edc0 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient: _finish_auth 0 2024-09-18T11:48:11.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 5 monclient: authenticate success, global_id 84167 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 10 monclient: _renew_subs 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1141420e0 con 0x7fb11412edc0 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 10 client.84167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 10 monclient: _renew_subs 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11412e880 con 0x7fb11412edc0 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 1 librados: init done 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1137fe700 1 --1- 172.21.15.112:0/2526394076 >> v1:172.21.15.112:6790/0 conn(0x7fb11412edc0 0x7fb11412e170 :-1 s=OPENED pgs=8827 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7fb0fc004c40 mon_map magic: 0 v1 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0fc0047f0 con 0x7fb11412edc0 2024-09-18T11:48:11.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1127fc700 4 set_mon_vals no callback set 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb1191c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.156+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb0fc004c40 con 0x7fb11412edc0 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.160+0000 7fb1191c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.160+0000 7fb1191c0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.160+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11405f2e0 con 0x7fb11412edc0 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.160+0000 7fb10bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.160+0000 7fb10bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.160+0000 7fb10bfff700 10 monclient: dump: 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:11.174 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:11.175 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:11.175 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:11.175 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:11.175 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:11.175 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:11.175 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:11.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.160+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb0fc004e20 con 0x7fb11412edc0 2024-09-18T11:48:11.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.160+0000 7fb113fff700 10 monclient: get_auth_request con 0x7fb0f4038070 auth_method 0 2024-09-18T11:48:11.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.164+0000 7fb10bfff700 10 client.84167.objecter ms_handle_connect 0x7fb0f4038070 2024-09-18T11:48:11.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.168+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(166..166 src has 1..166) v4 ==== 6835+0+0 (unknown 3653208750 0 0) 0x7fb0fc0444c0 con 0x7fb11412edc0 2024-09-18T11:48:11.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.168+0000 7fb10bfff700 10 client.84167.objecter ms_dispatch 0x7fb114000fd0 osd_map(166..166 src has 1..166) v4 2024-09-18T11:48:11.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.168+0000 7fb10bfff700 3 client.84167.objecter handle_osd_map got epochs [166,166] > 0 2024-09-18T11:48:11.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.168+0000 7fb10bfff700 3 client.84167.objecter handle_osd_map decoding full epoch 166 2024-09-18T11:48:11.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.168+0000 7fb10bfff700 20 client.84167.objecter dump_active .. 0 homeless 2024-09-18T11:48:11.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.988+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fb0fc005570 con 0x7fb11412edc0 2024-09-18T11:48:11.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.988+0000 7fb10bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:11.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:11.988+0000 7fb10bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:12.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:48:12.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:12.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:12.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb114063680 con 0x7fb11412edc0 2024-09-18T11:48:12.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb10bfff700 1 -- 172.21.15.112:0/2526394076 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2977136076) 0x7fb0fc014610 con 0x7fb11412edc0 2024-09-18T11:48:12.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb10bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:12.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb10bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:12.218 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:48:12.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 10 librados: watch_flush enter 2024-09-18T11:48:12.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 10 librados: watch_flush exit 2024-09-18T11:48:12.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 20 client.84167.objecter shutdown clearing up homeless session... 2024-09-18T11:48:12.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 10 client.84167.objecter successfully canceled tick 2024-09-18T11:48:12.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 >> v1:172.21.15.129:6812/20806 conn(0x7fb0f4038070 legacy=0x7fb0f403a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:12.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 10 monclient: shutdown 2024-09-18T11:48:12.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:12.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 >> v1:172.21.15.112:6790/0 conn(0x7fb11412edc0 legacy=0x7fb11412e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:12.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 shutdown_connections 2024-09-18T11:48:12.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 >> 172.21.15.112:0/2526394076 conn(0x7fb114067da0 msgr2=0x7fb1140a0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:12.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.212+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 shutdown_connections 2024-09-18T11:48:12.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.216+0000 7fb1191c0700 1 -- 172.21.15.112:0/2526394076 wait complete. 2024-09-18T11:48:12.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:12.216+0000 7fb1191c0700 1 librados: shutdown 2024-09-18T11:48:15.232 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient: get_monmap_and_config 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient: build_initial_monmap 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient: monmap: 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:15.345892+0000 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:15.345892+0000 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:15.346 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:15.347 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:15.347 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:15.347 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:15.347 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 1 Processor -- start 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 1 -- start start 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient: init 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient(hunting): picked mon.noname-c con 0x7ff0640a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient(hunting): picked mon.noname-a con 0x7ff06406c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient(hunting): picked mon.noname-b con 0x7ff06406c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:15.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06406e950 con 0x7ff06406c280 2024-09-18T11:48:15.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0640a2270 con 0x7ff0640a1ea0 2024-09-18T11:48:15.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06406d130 con 0x7ff06406c670 2024-09-18T11:48:15.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:15.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff06359e700 10 monclient(hunting): authenticate will time out at 1067.187102s 2024-09-18T11:48:15.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff059d9b700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7ff06406c280 legacy=0x7ff064135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:15.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff062d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff06406c670 0x7ff0641388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:38550/0 (socket says 172.21.15.112:38550) 2024-09-18T11:48:15.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.344+0000 7ff062d9d700 1 -- 172.21.15.112:0/4108917214 learned_addr learned my addr 172.21.15.112:0/4108917214 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:15.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.348+0000 7ff062d9d700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6790/0 conn(0x7ff06406c670 legacy=0x7ff0641388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:15.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.348+0000 7ff06259c700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.129:6789/0 conn(0x7ff0640a1ea0 legacy=0x7ff06406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:15.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.548+0000 7ff059d9b700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff06406c280 legacy=0x7ff064135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:15.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:15.952+0000 7ff059d9b700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff06406c280 legacy=0x7ff064135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:16.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:16.756+0000 7ff059d9b700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff06406c280 legacy=0x7ff064135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:18.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 10 monclient(hunting): tick 2024-09-18T11:48:18.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 1 monclient(hunting): continuing hunt 2024-09-18T11:48:18.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6790/0 conn(0x7ff06406c670 legacy=0x7ff0641388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.129:6789/0 conn(0x7ff0640a1ea0 legacy=0x7ff06406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff06406c280 legacy=0x7ff064135170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 10 monclient(hunting): picked mon.noname-a con 0x7ff044003810 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 10 monclient(hunting): picked mon.noname-b con 0x7ff06406c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 10 monclient(hunting): picked mon.noname-c con 0x7ff0640a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff06259c700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06409e370 con 0x7ff044003810 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06406e950 con 0x7ff0640a1ea0 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff044010530 con 0x7ff06406c670 2024-09-18T11:48:18.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.344+0000 7ff060d99700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:18.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.348+0000 7ff059d9b700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6790/0 conn(0x7ff06406c670 legacy=0x7ff0440094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:18.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.348+0000 7ff062d9d700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.129:6789/0 conn(0x7ff0640a1ea0 legacy=0x7ff04400cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:18.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.548+0000 7ff06259c700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:18.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:18.952+0000 7ff06259c700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:19.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:19.756+0000 7ff06259c700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 10 monclient(hunting): tick 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 1 monclient(hunting): continuing hunt 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6790/0 conn(0x7ff06406c670 legacy=0x7ff0440094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.129:6789/0 conn(0x7ff0640a1ea0 legacy=0x7ff04400cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 10 monclient(hunting): picked mon.noname-c con 0x7ff04400f5e0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 10 monclient(hunting): picked mon.noname-b con 0x7ff06406c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 10 monclient(hunting): picked mon.noname-a con 0x7ff044003810 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440117f0 con 0x7ff044003810 2024-09-18T11:48:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.344+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06409e370 con 0x7ff04400f5e0 2024-09-18T11:48:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.348+0000 7ff060d99700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440102d0 con 0x7ff06406c670 2024-09-18T11:48:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.348+0000 7ff062d9d700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.348+0000 7ff060d99700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.348+0000 7ff059d9b700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6790/0 conn(0x7ff06406c670 legacy=0x7ff04400cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.348+0000 7ff06259c700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.129:6789/0 conn(0x7ff04400f5e0 legacy=0x7ff04400f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:21.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.552+0000 7ff062d9d700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:21.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:21.956+0000 7ff062d9d700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:22.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.492+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080525922 0 0) 0x7ff06409e370 con 0x7ff04400f5e0 2024-09-18T11:48:22.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.492+0000 7ff061d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:22.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.492+0000 7ff061d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:22.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.492+0000 7ff061d9b700 10 monclient(hunting): my global_id is 17893 2024-09-18T11:48:22.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.492+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff050003570 con 0x7ff04400f5e0 2024-09-18T11:48:22.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.500+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899834338 0 0) 0x7ff044010530 con 0x7ff06406c670 2024-09-18T11:48:22.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.500+0000 7ff061d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:22.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.500+0000 7ff061d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:22.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.500+0000 7ff061d9b700 10 monclient(hunting): my global_id is 84170 2024-09-18T11:48:22.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.500+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff06409e370 con 0x7ff06406c670 2024-09-18T11:48:22.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1488133420 0 0) 0x7ff050003570 con 0x7ff04400f5e0 2024-09-18T11:48:22.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6790/0 conn(0x7ff06406c670 legacy=0x7ff04400cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.112:6789/0 conn(0x7ff044003810 legacy=0x7ff044005c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:48:22.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:48:22.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 1 monclient: found mon.noname-c 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0440085a0 con 0x7ff04400f5e0 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: _finish_auth 0 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff06359e700 5 monclient: authenticate success, global_id 17893 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff04c0019b0 con 0x7ff04400f5e0 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: dump: 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.542 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.543 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.543 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.543 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.543 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.543 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.543 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.543 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff06359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:22.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff04c001c40 con 0x7ff04400f5e0 2024-09-18T11:48:22.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:22.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 1 -- 172.21.15.112:0/4108917214 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff04c0020a0 con 0x7ff04400f5e0 2024-09-18T11:48:22.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:22.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff061d9b700 10 monclient: dump: 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff06359e700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:22.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.540+0000 7ff06359e700 4 set_mon_vals no callback set 2024-09-18T11:48:22.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 10 monclient: shutdown 2024-09-18T11:48:22.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:22.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 -- 172.21.15.112:0/4108917214 >> v1:172.21.15.129:6789/0 conn(0x7ff04400f5e0 legacy=0x7ff04400f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 -- 172.21.15.112:0/4108917214 shutdown_connections 2024-09-18T11:48:22.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 -- 172.21.15.112:0/4108917214 >> 172.21.15.112:0/4108917214 conn(0x7ff064067da0 msgr2=0x7ff0640a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:22.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 -- 172.21.15.112:0/4108917214 shutdown_connections 2024-09-18T11:48:22.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 -- 172.21.15.112:0/4108917214 wait complete. 2024-09-18T11:48:22.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 10 monclient: build_initial_monmap 2024-09-18T11:48:22.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 10 monclient: monmap: 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 librados: starting msgr at 2024-09-18T11:48:22.548 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 librados: starting objecter 2024-09-18T11:48:22.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 Processor -- start 2024-09-18T11:48:22.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 -- start start 2024-09-18T11:48:22.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 librados: setting wanted keys 2024-09-18T11:48:22.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 1 librados: calling monclient init 2024-09-18T11:48:22.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.544+0000 7ff06359e700 10 monclient: init 2024-09-18T11:48:22.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 10 monclient(hunting): picked mon.noname-b con 0x7ff044003810 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 10 monclient(hunting): picked mon.noname-c con 0x7ff04400f5e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 10 monclient(hunting): picked mon.noname-a con 0x7ff06406c670 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff064121570 con 0x7ff06406c670 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff064120430 con 0x7ff044003810 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff062d9d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7ff06406c670 legacy=0x7ff064149880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff064121290 con 0x7ff04400f5e0 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06359e700 10 monclient(hunting): authenticate will time out at 1074.389089s 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff059d9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff04400f5e0 0x7ff064132150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37872/0 (socket says 172.21.15.112:37872) 2024-09-18T11:48:22.551 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff059d9b700 1 -- 172.21.15.112:0/2866390319 learned_addr learned my addr 172.21.15.112:0/2866390319 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:22.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06259c700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff044003810 0x7ff0641414f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.112:58888/0 (socket says 172.21.15.112:58888) 2024-09-18T11:48:22.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff059d9b700 1 -- 172.21.15.112:0/2866390319 >> v1:172.21.15.112:6790/0 conn(0x7ff04400f5e0 legacy=0x7ff064132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff05affd700 10 client.?.objecter ms_handle_connect 0x7ff04400f5e0 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff05affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff05affd700 10 client.?.objecter ms_handle_connect 0x7ff044003810 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff05affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.548+0000 7ff06259c700 1 -- 172.21.15.112:0/2866390319 >> v1:172.21.15.129:6789/0 conn(0x7ff044003810 legacy=0x7ff0641414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159739550 0 0) 0x7ff064120430 con 0x7ff044003810 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient(hunting): my global_id is 17896 2024-09-18T11:48:22.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff044011790 con 0x7ff044003810 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3915822336 0 0) 0x7ff044011790 con 0x7ff044003810 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff064120430 con 0x7ff044003810 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff04c0020a0 con 0x7ff044003810 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient(hunting): dump: 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.555 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 811216973 0 0) 0x7ff064120430 con 0x7ff044003810 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 >> v1:172.21.15.112:6790/0 conn(0x7ff04400f5e0 legacy=0x7ff064132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 >> v1:172.21.15.112:6789/0 conn(0x7ff06406c670 legacy=0x7ff064149880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient: _finish_hunting 0 2024-09-18T11:48:22.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 monclient: found mon.b 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff064132dd0 con 0x7ff044003810 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient: _finish_auth 0 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 5 monclient: authenticate success, global_id 17896 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 10 monclient: _renew_subs 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff064141c00 con 0x7ff044003810 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 10 client.17896.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 10 monclient: _renew_subs 2024-09-18T11:48:22.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:22.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff064132a00 con 0x7ff044003810 2024-09-18T11:48:22.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 1 librados: init done 2024-09-18T11:48:22.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff04c001570 con 0x7ff044003810 2024-09-18T11:48:22.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:22.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff04c002710 con 0x7ff044003810 2024-09-18T11:48:22.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06159a700 4 set_mon_vals no callback set 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff05affd700 10 monclient: dump: 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.559 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.552+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0480050b0 con 0x7ff044003810 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.556+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ff04c002970 con 0x7ff044003810 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.556+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(166..166 src has 1..166) v4 ==== 6835+0+0 (unknown 3653208750 0 0) 0x7ff04c044450 con 0x7ff044003810 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.556+0000 7ff05affd700 10 client.17896.objecter ms_dispatch 0x7ff064000fd0 osd_map(166..166 src has 1..166) v4 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.556+0000 7ff05affd700 3 client.17896.objecter handle_osd_map got epochs [166,166] > 0 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.556+0000 7ff05affd700 3 client.17896.objecter handle_osd_map decoding full epoch 166 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.556+0000 7ff059d9b700 10 monclient: get_auth_request con 0x7ff0440390c0 auth_method 0 2024-09-18T11:48:22.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.556+0000 7ff05affd700 20 client.17896.objecter dump_active .. 0 homeless 2024-09-18T11:48:22.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.556+0000 7ff05affd700 10 client.17896.objecter ms_handle_connect 0x7ff0440390c0 2024-09-18T11:48:22.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.568+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7ff04c014b40 con 0x7ff044003810 2024-09-18T11:48:22.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.568+0000 7ff05affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:22.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.568+0000 7ff05affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:22.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.808+0000 7ff06359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:48:22.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.808+0000 7ff06359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:22.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.808+0000 7ff06359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:48:22.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.808+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff0480058e0 con 0x7ff044003810 2024-09-18T11:48:22.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff05affd700 1 -- 172.21.15.112:0/2866390319 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 455013894) 0x7ff04c014540 con 0x7ff044003810 2024-09-18T11:48:22.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff05affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:48:22.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff05affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:22.830 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:48:22.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 10 librados: watch_flush enter 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 10 librados: watch_flush exit 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 20 client.17896.objecter shutdown clearing up homeless session... 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 10 client.17896.objecter successfully canceled tick 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 >> v1:172.21.15.129:6812/20806 conn(0x7ff0440390c0 legacy=0x7ff04403b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 10 monclient: shutdown 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 >> v1:172.21.15.129:6789/0 conn(0x7ff044003810 legacy=0x7ff0641414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 shutdown_connections 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.828+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 >> 172.21.15.112:0/2866390319 conn(0x7ff064067da0 msgr2=0x7ff064099460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.832+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 shutdown_connections 2024-09-18T11:48:22.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.832+0000 7ff06359e700 1 -- 172.21.15.112:0/2866390319 wait complete. 2024-09-18T11:48:22.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.832+0000 7ff06359e700 1 librados: shutdown 2024-09-18T11:48:22.846 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T11:48:22.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T11:48:22.846 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:48:22.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.948+0000 7f8ecb59e700 10 monclient: get_monmap_and_config 2024-09-18T11:48:22.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.948+0000 7f8ecb59e700 10 monclient: build_initial_monmap 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.948+0000 7f8ecb59e700 10 monclient: monmap: 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:22.951308+0000 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:22.951308+0000 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:22.952 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.948+0000 7f8ecb59e700 1 Processor -- start 2024-09-18T11:48:22.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.948+0000 7f8ecb59e700 1 -- start start 2024-09-18T11:48:22.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.948+0000 7f8ecb59e700 10 monclient: init 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.948+0000 7f8ecb59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.948+0000 7f8ecb59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 10 monclient(hunting): picked mon.noname-a con 0x7f8ecc0a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 10 monclient(hunting): picked mon.noname-c con 0x7f8ecc06c280 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 10 monclient(hunting): picked mon.noname-b con 0x7f8ecc06c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc06e950 con 0x7f8ecc0a1ea0 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc0a2270 con 0x7f8ecc06c280 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc06d130 con 0x7f8ecc06c670 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 10 monclient(hunting): authenticate will time out at 1074.792112s 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8eca59c700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f8ecc0a1ea0 legacy=0x7f8ecc06bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecad9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8ecc06c670 0x7f8ecc1388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37880/0 (socket says 172.21.15.112:37880) 2024-09-18T11:48:22.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecad9d700 1 -- 172.21.15.112:0/3281881891 learned_addr learned my addr 172.21.15.112:0/3281881891 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:22.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecad9d700 1 -- 172.21.15.112:0/3281881891 >> v1:172.21.15.112:6790/0 conn(0x7f8ecc06c670 legacy=0x7f8ecc1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:22.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555909485 0 0) 0x7f8ecc06d130 con 0x7f8ecc06c670 2024-09-18T11:48:22.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:22.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient(hunting): my global_id is 84179 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eac003570 con 0x7f8ecc06c670 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec9d9b700 1 -- 172.21.15.112:0/3281881891 >> v1:172.21.15.129:6789/0 conn(0x7f8ecc06c280 legacy=0x7f8ecc135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 998763394 0 0) 0x7f8eac003570 con 0x7f8ecc06c670 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 >> v1:172.21.15.129:6789/0 conn(0x7f8ecc06c280 legacy=0x7f8ecc135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 >> v1:172.21.15.112:6789/0 conn(0x7f8ecc0a1ea0 legacy=0x7f8ecc06bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: _finish_hunting 0 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 monclient: found mon.noname-b 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc09e370 con 0x7f8ecc06c670 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: _finish_auth 0 2024-09-18T11:48:22.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 5 monclient: authenticate success, global_id 84179 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8ec4001d80 con 0x7f8ecc06c670 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: dump: 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.958 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ec4001ee0 con 0x7f8ecc06c670 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 1 -- 172.21.15.112:0/3281881891 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8ec4004c70 con 0x7f8ecc06c670 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 4 set_mon_vals no callback set 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ec959a700 10 monclient: dump: 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 10 monclient: shutdown 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:22.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 1 -- 172.21.15.112:0/3281881891 >> v1:172.21.15.112:6790/0 conn(0x7f8ecc06c670 legacy=0x7f8ecc1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 1 -- 172.21.15.112:0/3281881891 shutdown_connections 2024-09-18T11:48:22.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.952+0000 7f8ecb59e700 1 -- 172.21.15.112:0/3281881891 >> 172.21.15.112:0/3281881891 conn(0x7f8ecc067da0 msgr2=0x7f8ecc0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:22.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 -- 172.21.15.112:0/3281881891 shutdown_connections 2024-09-18T11:48:22.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 -- 172.21.15.112:0/3281881891 wait complete. 2024-09-18T11:48:22.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient: build_initial_monmap 2024-09-18T11:48:22.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient: monmap: 2024-09-18T11:48:22.964 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 librados: starting msgr at 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 librados: starting objecter 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 Processor -- start 2024-09-18T11:48:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 -- start start 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 librados: setting wanted keys 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 librados: calling monclient init 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient: init 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient(hunting): picked mon.noname-a con 0x7f8ecc06c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient(hunting): picked mon.noname-c con 0x7f8ecc06c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient(hunting): picked mon.noname-b con 0x7f8ecc0a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc06e1a0 con 0x7f8ecc06c280 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc09d860 con 0x7f8ecc0a1ea0 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc139940 con 0x7f8ecc06c670 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8eca59c700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f8ecc06c280 legacy=0x7f8ecc139230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:22.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ecb59e700 10 monclient(hunting): authenticate will time out at 1074.798043s 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ec9d9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8ecc06c670 0x7f8ecc1468a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37890/0 (socket says 172.21.15.112:37890) 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.956+0000 7f8ec9d9b700 1 -- 172.21.15.112:0/785684092 learned_addr learned my addr 172.21.15.112:0/785684092 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec9d9b700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6790/0 conn(0x7f8ecc06c670 legacy=0x7f8ecc1468a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 client.?.objecter ms_handle_connect 0x7f8ecc06c670 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081556911 0 0) 0x7f8ecc139940 con 0x7f8ecc06c670 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecad9d700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.129:6789/0 conn(0x7f8ecc0a1ea0 legacy=0x7f8ecc149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): my global_id is 84182 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb0004f10 con 0x7f8ecc06c670 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 client.?.objecter ms_handle_connect 0x7f8ecc0a1ea0 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:22.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521892910 0 0) 0x7f8ecc09d860 con 0x7f8ecc0a1ea0 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): my global_id is 17908 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ecc139940 con 0x7f8ecc0a1ea0 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809225445 0 0) 0x7f8eb0004f10 con 0x7f8ecc06c670 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ecc09d860 con 0x7f8ecc06c670 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8eb4001c00 con 0x7f8ecc06c670 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): dump: 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.968 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1119119564 0 0) 0x7f8ecc09d860 con 0x7f8ecc06c670 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.129:6789/0 conn(0x7f8ecc0a1ea0 legacy=0x7f8ecc149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6789/0 conn(0x7f8ecc06c280 legacy=0x7f8ecc139230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient: _finish_hunting 0 2024-09-18T11:48:22.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 monclient: found mon.c 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc11c810 con 0x7f8ecc06c670 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient: _finish_auth 0 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 5 monclient: authenticate success, global_id 84182 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 10 monclient: _renew_subs 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc11e2d0 con 0x7f8ecc06c670 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 10 client.84182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 10 monclient: _renew_subs 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:22.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc11d560 con 0x7f8ecc06c670 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 1 librados: init done 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8eb40022b0 con 0x7f8ecc06c670 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8eb4002600 con 0x7f8ecc06c670 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec8d99700 4 set_mon_vals no callback set 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 monclient: dump: 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.971 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8ea4006590 con 0x7f8ea4002840 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8eb4002760 con 0x7f8ecc06c670 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(167..167 src has 1..167) v4 ==== 6483+0+0 (unknown 3490533639 0 0) 0x7f8eb40442c0 con 0x7f8ecc06c670 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 10 client.84182.objecter ms_dispatch 0x7f8ecc000fd0 osd_map(167..167 src has 1..167) v4 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 3 client.84182.objecter handle_osd_map got epochs [167,167] > 0 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 3 client.84182.objecter handle_osd_map decoding full epoch 167 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ec27fc700 20 client.84182.objecter dump_active .. 0 homeless 2024-09-18T11:48:22.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.960+0000 7f8ecad9d700 10 monclient: get_auth_request con 0x7f8eb00381b0 auth_method 0 2024-09-18T11:48:22.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 client.84182.objecter ms_handle_connect 0x7f8ea4002840 2024-09-18T11:48:22.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 client.84182.objecter resend_mon_ops 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8eca59c700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6790/0 conn(0x7f8ea4002840 legacy=0x7f8ea4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686398477 0 0) 0x7f8ea4006590 con 0x7f8ea4002840 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 monclient(hunting): my global_id is 84185 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ecc09d860 con 0x7f8ea4002840 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2453075850 0 0) 0x7f8ecc09d860 con 0x7f8ea4002840 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ea4006590 con 0x7f8ea4002840 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8ebc0043a0 con 0x7f8ea4002840 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 monclient: dump: 2024-09-18T11:48:22.975 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1226332832 0 0) 0x7f8ea4006590 con 0x7f8ea4002840 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ea400a140 con 0x7f8ea4002840 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 client.84182.objecter ms_handle_connect 0x7f8eb00381b0 2024-09-18T11:48:22.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f8ebc005810 con 0x7f8ea4002840 2024-09-18T11:48:22.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:22.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:22.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6790/0 conn(0x7f8ea4002840 legacy=0x7f8ea4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.964+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6790/0 conn(0x7f8ea4002840 legacy=0x7f8ea4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:22.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ecb59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:48:22.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8ea4009670 con 0x7f8ea40090e0 2024-09-18T11:48:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8eca59c700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6790/0 conn(0x7f8ea40090e0 legacy=0x7f8ea4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 10 client.84182.objecter ms_handle_connect 0x7f8ea40090e0 2024-09-18T11:48:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 10 client.84182.objecter resend_mon_ops 2024-09-18T11:48:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831416489 0 0) 0x7f8ea4009670 con 0x7f8ea40090e0 2024-09-18T11:48:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 10 monclient(hunting): my global_id is 84188 2024-09-18T11:48:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea4006590 con 0x7f8ea40090e0 2024-09-18T11:48:22.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1941433620 0 0) 0x7f8ea4006590 con 0x7f8ea40090e0 2024-09-18T11:48:22.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ea4009670 con 0x7f8ea40090e0 2024-09-18T11:48:22.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8ebc004950 con 0x7f8ea40090e0 2024-09-18T11:48:22.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.976+0000 7f8ec27fc700 10 monclient: dump: 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:22.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3223348365 0 0) 0x7f8ea4009670 con 0x7f8ea40090e0 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8ea4009930 con 0x7f8ea40090e0 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3990 (unknown 0 0 637412433) 0x7f8ebc005aa0 con 0x7f8ea40090e0 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ec27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ec27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6790/0 conn(0x7f8ea40090e0 legacy=0x7f8ea4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ec27fc700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6790/0 conn(0x7f8ea40090e0 legacy=0x7f8ea4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:22.983 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 24, 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:22.984 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:22.985 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:48:22.986 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:48:22.987 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.988 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:22.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:22.990 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:22.991 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:48:22.995 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:48:22.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 10 librados: watch_flush enter 2024-09-18T11:48:22.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 10 librados: watch_flush exit 2024-09-18T11:48:22.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 20 client.84182.objecter shutdown clearing up homeless session... 2024-09-18T11:48:22.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 10 client.84182.objecter successfully canceled tick 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.129:6812/20806 conn(0x7f8eb00381b0 legacy=0x7f8eb003a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 10 monclient: shutdown 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 >> v1:172.21.15.112:6790/0 conn(0x7f8ecc06c670 legacy=0x7f8ecc1468a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecad9d700 1 -- 172.21.15.112:0/785684092 reap_dead start 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 shutdown_connections 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 >> 172.21.15.112:0/785684092 conn(0x7f8ecc067da0 msgr2=0x7f8ecc09fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 shutdown_connections 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 1 -- 172.21.15.112:0/785684092 wait complete. 2024-09-18T11:48:22.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:22.980+0000 7f8ecb59e700 1 librados: shutdown 2024-09-18T11:48:22.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:48:22.998 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T11:48:22.998 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient: get_monmap_and_config 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient: build_initial_monmap 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient: monmap: 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:48:23.109770+0000 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:48:23.109770+0000 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:23.110 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:23.111 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:48:23.111 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:48:23.111 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 1 Processor -- start 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 1 -- start start 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient: init 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient(hunting): picked mon.noname-b con 0x7f08a006add0 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient(hunting): picked mon.noname-c con 0x7f08a0132c30 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient(hunting): picked mon.noname-a con 0x7f08a0133070 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:23.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a00a17f0 con 0x7f08a0133070 2024-09-18T11:48:23.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a0097010 con 0x7f08a0132c30 2024-09-18T11:48:23.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a00972f0 con 0x7f08a006add0 2024-09-18T11:48:23.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:23.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f08a4f05700 10 monclient(hunting): authenticate will time out at 1074.950946s 2024-09-18T11:48:23.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f089ed9d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f08a0133070 legacy=0x7f08a013af80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:23.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f089e59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f08a006add0 0x7f08a0132520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37916/0 (socket says 172.21.15.112:37916) 2024-09-18T11:48:23.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.108+0000 7f089e59c700 1 -- 172.21.15.112:0/2086451826 learned_addr learned my addr 172.21.15.112:0/2086451826 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:23.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089e59c700 1 -- 172.21.15.112:0/2086451826 >> v1:172.21.15.112:6790/0 conn(0x7f08a006add0 legacy=0x7f08a0132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:23.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089dd9b700 1 -- 172.21.15.112:0/2086451826 >> v1:172.21.15.129:6789/0 conn(0x7f08a0132c30 legacy=0x7f08a0137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:23.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117367133 0 0) 0x7f08a00972f0 con 0x7f08a006add0 2024-09-18T11:48:23.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:23.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient(hunting): my global_id is 84191 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0888003570 con 0x7f08a006add0 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027763729 0 0) 0x7f08a0097010 con 0x7f08a0132c30 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient(hunting): my global_id is 17911 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a00972f0 con 0x7f08a0132c30 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3163879730 0 0) 0x7f0888003570 con 0x7f08a006add0 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 >> v1:172.21.15.129:6789/0 conn(0x7f08a0132c30 legacy=0x7f08a0137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 >> v1:172.21.15.112:6789/0 conn(0x7f08a0133070 legacy=0x7f08a013af80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: _finish_hunting 0 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 monclient: found mon.noname-b 2024-09-18T11:48:23.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a00975d0 con 0x7f08a006add0 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: _finish_auth 0 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 5 monclient: authenticate success, global_id 84191 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0894001940 con 0x7f08a006add0 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: dump: 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:23.117 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0894001bd0 con 0x7f08a006add0 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 10 monclient: get_monmap_and_config success 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 4 set_mon_vals no callback set 2024-09-18T11:48:23.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 1 -- 172.21.15.112:0/2086451826 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0894004a90 con 0x7f08a006add0 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f089d59a700 10 monclient: dump: 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:23.119 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:23.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 10 monclient: shutdown 2024-09-18T11:48:23.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:23.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 1 -- 172.21.15.112:0/2086451826 >> v1:172.21.15.112:6790/0 conn(0x7f08a006add0 legacy=0x7f08a0132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:23.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 1 -- 172.21.15.112:0/2086451826 shutdown_connections 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 1 -- 172.21.15.112:0/2086451826 >> 172.21.15.112:0/2086451826 conn(0x7f08a00a5f90 msgr2=0x7f08a00a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 1 -- 172.21.15.112:0/2086451826 shutdown_connections 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.112+0000 7f08a4f05700 1 -- 172.21.15.112:0/2086451826 wait complete. 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient: build_initial_monmap 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient: monmap: 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:23.124 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 librados: starting msgr at 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 librados: starting objecter 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 Processor -- start 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 -- start start 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 librados: setting wanted keys 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 librados: calling monclient init 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient: init 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient: _reopen_session rank -1 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient(hunting): picked mon.noname-b con 0x7f08a006add0 addr v1:172.21.15.129:6789/0 2024-09-18T11:48:23.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient(hunting): picked mon.noname-c con 0x7f08a0132c30 addr v1:172.21.15.112:6790/0 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient(hunting): picked mon.noname-a con 0x7f08a0133070 addr v1:172.21.15.112:6789/0 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a011ee50 con 0x7f08a0133070 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a011eae0 con 0x7f08a006add0 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a011e800 con 0x7f08a0132c30 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient(hunting): _renew_subs 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f089ed9d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f08a0133070 legacy=0x7f08a014a360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f08a4f05700 10 monclient(hunting): authenticate will time out at 1074.958189s 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f089dd9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f08a0132c30 0x7f08a0146c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37920/0 (socket says 172.21.15.112:37920) 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.116+0000 7f089dd9b700 1 -- 172.21.15.112:0/4214070382 learned_addr learned my addr 172.21.15.112:0/4214070382 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f089dd9b700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.112:6790/0 conn(0x7f08a0132c30 legacy=0x7f08a0146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 client.?.objecter ms_handle_connect 0x7f08a0132c30 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f089e59c700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6789/0 conn(0x7f08a006add0 legacy=0x7f08a01316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:23.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 client.?.objecter ms_handle_connect 0x7f08a006add0 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685834707 0 0) 0x7f08a011e800 con 0x7f08a0132c30 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): my global_id is 84194 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0880004f10 con 0x7f08a0132c30 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479201716 0 0) 0x7f08a011eae0 con 0x7f08a006add0 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): my global_id is 17914 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a011e800 con 0x7f08a006add0 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2513180882 0 0) 0x7f0880004f10 con 0x7f08a0132c30 2024-09-18T11:48:23.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a011eae0 con 0x7f08a0132c30 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0890002060 con 0x7f08a0132c30 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): dump: 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:23.128 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3776001763 0 0) 0x7f08a011eae0 con 0x7f08a0132c30 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6789/0 conn(0x7f08a006add0 legacy=0x7f08a01316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.112:6789/0 conn(0x7f08a0133070 legacy=0x7f08a014a360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 monclient: found mon.c 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a0131df0 con 0x7f08a0132c30 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:48:23.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 5 monclient: authenticate success, global_id 84194 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 10 monclient: _renew_subs 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a011d8a0 con 0x7f08a0132c30 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 10 client.84194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 10 monclient: _renew_subs 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a0141f70 con 0x7f08a0132c30 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 1 librados: init done 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0890002680 con 0x7f08a0132c30 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0890002ad0 con 0x7f08a0132c30 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:23.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 10 monclient: dump: 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f089cd99700 4 set_mon_vals no callback set 2024-09-18T11:48:23.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f08a4f05700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:23.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.120+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0890002cf0 con 0x7f08a0132c30 2024-09-18T11:48:23.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(167..167 src has 1..167) v4 ==== 6483+0+0 (unknown 3490533639 0 0) 0x7f0890044730 con 0x7f08a0132c30 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 client.84194.objecter ms_dispatch 0x7f08a0000fd0 osd_map(167..167 src has 1..167) v4 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 3 client.84194.objecter handle_osd_map got epochs [167,167] > 0 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 3 client.84194.objecter handle_osd_map decoding full epoch 167 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f08a009dc40 con 0x7f08a009d870 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 20 client.84194.objecter dump_active .. 0 homeless 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f089e59c700 10 monclient: get_auth_request con 0x7f0880037f20 auth_method 0 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f089ed9d700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6789/0 conn(0x7f08a009d870 legacy=0x7f08a005e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 client.84194.objecter ms_handle_connect 0x7f08a009d870 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 client.84194.objecter resend_mon_ops 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 client.84194.objecter ms_handle_connect 0x7f0880037f20 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025935113 0 0) 0x7f08a009dc40 con 0x7f08a009d870 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:23.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 monclient(hunting): my global_id is 17917 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a011eae0 con 0x7f08a009d870 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2941148397 0 0) 0x7f08a011eae0 con 0x7f08a009d870 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a009dc40 con 0x7f08a009d870 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0898004210 con 0x7f08a009d870 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.124+0000 7f088e7fc700 10 monclient: dump: 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.135 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.128+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1710364488 0 0) 0x7f08a009dc40 con 0x7f08a009d870 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.128+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08a005f2e0 con 0x7f08a009d870 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.128+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f0898006560 con 0x7f08a009d870 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.128+0000 7f088e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:48:23.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.128+0000 7f088e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:48:23.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.128+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6789/0 conn(0x7f08a009d870 legacy=0x7f08a005e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:23.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.128+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6789/0 conn(0x7f08a009d870 legacy=0x7f08a005e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:23.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.140+0000 7f08a4f05700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:48:23.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.140+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f08a002b280 con 0x7f08a002ae80 2024-09-18T11:48:23.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f089ed9d700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6789/0 conn(0x7f08a002ae80 legacy=0x7f08a00998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:48:23.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 client.84194.objecter ms_handle_connect 0x7f08a002ae80 2024-09-18T11:48:23.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 client.84194.objecter resend_mon_ops 2024-09-18T11:48:23.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450709710 0 0) 0x7f08a002b280 con 0x7f08a002ae80 2024-09-18T11:48:23.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:48:23.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:48:23.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 monclient(hunting): my global_id is 17920 2024-09-18T11:48:23.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a009dc40 con 0x7f08a002ae80 2024-09-18T11:48:23.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3181685352 0 0) 0x7f08a009dc40 con 0x7f08a002ae80 2024-09-18T11:48:23.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a002b280 con 0x7f08a002ae80 2024-09-18T11:48:23.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0898004a20 con 0x7f08a002ae80 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 monclient: dump: 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:48:23.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3017603199 0 0) 0x7f08a002b280 con 0x7f08a002ae80 2024-09-18T11:48:23.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f08a0063a80 con 0x7f08a002ae80 2024-09-18T11:48:23.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4162 (unknown 0 0 4091816740) 0x7f0898005e60 con 0x7f08a002ae80 2024-09-18T11:48:23.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:48:23.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:48:23.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6789/0 conn(0x7f08a002ae80 legacy=0x7f08a00998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.144+0000 7f088e7fc700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6789/0 conn(0x7f08a002ae80 legacy=0x7f08a00998c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 24, 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:48:23.150 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:23.151 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:23.152 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:48:23.153 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:48:23.154 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.155 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:48:23.156 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:23.158 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:23.160 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 7 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:48:23.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:48:23.162 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:48:23.162 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:48:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 10 librados: watch_flush enter 2024-09-18T11:48:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 10 librados: watch_flush exit 2024-09-18T11:48:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 20 client.84194.objecter shutdown clearing up homeless session... 2024-09-18T11:48:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 10 client.84194.objecter successfully canceled tick 2024-09-18T11:48:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.129:6812/20806 conn(0x7f0880037f20 legacy=0x7f088003a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 10 monclient: shutdown 2024-09-18T11:48:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:48:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 >> v1:172.21.15.112:6790/0 conn(0x7f08a0132c30 legacy=0x7f08a0146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:48:23.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f089ed9d700 1 -- 172.21.15.112:0/4214070382 reap_dead start 2024-09-18T11:48:23.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 shutdown_connections 2024-09-18T11:48:23.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 >> 172.21.15.112:0/4214070382 conn(0x7f08a00a5f90 msgr2=0x7f08a00a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:48:23.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 shutdown_connections 2024-09-18T11:48:23.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 1 -- 172.21.15.112:0/4214070382 wait complete. 2024-09-18T11:48:23.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:48:23.148+0000 7f08a4f05700 1 librados: shutdown 2024-09-18T11:48:23.166 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:48:23.166 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T11:48:23.583 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAii_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2024-09-18T11:48:23.583 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=9 expected=9 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=12 expected=12 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=4 expected=4 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2024-09-18T11:48:23.584 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=0 expected=0 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=14 expected=14 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=7 expected=7 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=1 expected=1 2024-09-18T11:48:23.585 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 7:566253c9:::13:head 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=13 expected=13 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (655 ms) 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (111621 ms) 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2024-09-18T11:48:23.586 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (45995 ms) 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 7 tests from LibRadosList (158812 ms total) 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (587 ms) 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo2,foo3 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo2 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo3 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo4,foo5 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo4 2024-09-18T11:48:23.587 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo5 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo6,foo7 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo7 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo6 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo4 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo5 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo7 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo6 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo1 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo1 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo2 2024-09-18T11:48:23.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo3 2024-09-18T11:48:23.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (101 ms) 2024-09-18T11:48:23.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-18T11:48:23.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 1 0 2024-09-18T11:48:23.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 10 0 2024-09-18T11:48:23.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 13 0 2024-09-18T11:48:29.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 7 0 2024-09-18T11:48:29.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 14 0 2024-09-18T11:48:29.287 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 0 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 15 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 11 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 5 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 8 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 6 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 3 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 4 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 12 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 9 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2 0 2024-09-18T11:48:29.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (163 ms) 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 3 tests from LibRadosListEC (851 ms total) 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2900 ms) 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 1 test from LibRadosListNP (2900 ms total) 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] Global test environment tear-down 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [==========] 11 tests from 3 test suites ran. (185386 ms total) 2024-09-18T11:48:29.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ PASSED ] 11 tests. 2024-09-18T11:49:00.307 INFO:tasks.workunit.client.0.smithi129.stdout:ete 2024-09-18T11:49:00.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6984 ms) 2024-09-18T11:49:00.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-18T11:49:00.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7033 ms) 2024-09-18T11:49:00.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-18T11:49:00.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7003 ms) 2024-09-18T11:49:00.308 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-18T11:49:00.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6999 ms) 2024-09-18T11:49:00.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-18T11:49:00.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7009 ms) 2024-09-18T11:49:00.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-18T11:49:00.309 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6991 ms) 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7005 ms) 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6989 ms) 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7468 ms) 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (20862 ms) 2024-09-18T11:49:00.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-18T11:49:00.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6926 ms) 2024-09-18T11:49:00.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-18T11:49:00.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7006 ms) 2024-09-18T11:49:00.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-18T11:49:00.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7410 ms) 2024-09-18T11:49:00.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-18T11:49:00.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6588 ms) 2024-09-18T11:49:00.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-18T11:49:00.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7004 ms) 2024-09-18T11:49:00.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (126262 ms total) 2024-09-18T11:49:00.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: 2024-09-18T11:49:00.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] Global test environment tear-down 2024-09-18T11:49:00.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (216456 ms total) 2024-09-18T11:49:00.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ PASSED ] 41 tests. 2024-09-18T11:49:00.314 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:49:00.314 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24309 2024-09-18T11:49:00.314 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24309 2024-09-18T11:49:00.314 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23831 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23831 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24299 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24299 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24199 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24199 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23776 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23776 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23982 2024-09-18T11:49:00.315 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23982 2024-09-18T11:49:00.316 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:49:00.316 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24109 2024-09-18T11:49:00.316 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24109 2024-09-18T11:49:00.316 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T11:49:00.316 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23961 2024-09-18T11:49:00.316 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23961 2024-09-18T11:49:08.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-18T11:49:08.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-18T11:49:08.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-18T11:49:08.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: seed 23975 2024-09-18T11:49:08.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (123 ms) 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10921 ms) 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (35 ms) 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11080 ms total) 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7114 ms) 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-18T11:49:08.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7893 ms) 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9908 ms) 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [4] 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [5,4] 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [6,5,4] 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting some heads 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 7 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-18T11:49:08.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrubs... 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47769 ms) 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10011 ms) 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8021 ms) 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7962 ms) 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8931 ms) 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9064 ms) 2024-09-18T11:49:08.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9075 ms) 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9978 ms) 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (27716 ms) 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7498 ms) 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (9010 ms) 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13038 ms) 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-18T11:49:08.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7961 ms) 2024-09-18T11:49:08.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-18T11:49:08.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (7993 ms) 2024-09-18T11:49:24.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: ok, hit_set contains 186:602f83fe:::foo:head 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9077 ms) 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg_num = 32 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 0 ls 1726660164,0 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 1 ls 1726660164,0 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 2 ls 1726660164,0 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 3 ls 1726660164,0 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 4 ls 1726660164,0 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 5 ls 1726660164,0 2024-09-18T11:49:24.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 6 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 7 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 8 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 9 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 10 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 11 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 12 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 13 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 14 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 15 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 16 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 17 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 18 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 19 ls 1726660164,0 2024-09-18T11:49:24.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 20 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 21 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 22 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 23 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 24 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 25 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 26 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 27 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 28 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 29 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 30 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 31 ls 1726660164,0 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg_num = 32 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:6cac518f:::0:head 2024-09-18T11:49:24.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:02547ec2:::1:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f905c69b:::2:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:cfc208b3:::3:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d83876eb:::4:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b29083e3:::5:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c4fdafeb:::6:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:5c6b0b28:::7:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:bd63b0f1:::8:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:e960b815:::9:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:52ea6a34:::10:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:89d3ae78:::11:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:de5d7c5f:::12:head 2024-09-18T11:49:24.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:566253c9:::13:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:62a1935d:::14:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:863748b0:::15:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:3958e169:::16:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:4d4dabf9:::17:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:8391935d:::18:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:28883081:::19:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:69259c59:::20:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:4bdb80b7:::21:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a11c5d71:::22:head 2024-09-18T11:49:24.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:271af37b:::23:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:95b121be:::24:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:58d1031b:::25:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:0a050783:::26:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c709704c:::27:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:cbe56eaf:::28:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:86b4b162:::29:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:70d89383:::30:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:dd450c7c:::31:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:6d5729b1:::32:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c388f3fb:::33:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:56cfea31:::34:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:9dbc1bf7:::35:head 2024-09-18T11:49:24.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:40b74ccd:::36:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:4d5aaf42:::37:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:920f362c:::38:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:6cc53222:::39:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:9cad833f:::40:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:1ea84d41:::41:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c4480ef6:::42:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a694361e:::43:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d1bd33e9:::44:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:ddc2cd5d:::45:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:2b782207:::46:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:7b187fca:::47:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:90ecdf6f:::48:head 2024-09-18T11:49:24.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a5ed95fe:::49:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:ea0eaa55:::50:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f33ef17b:::51:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a0d1b2f6:::52:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:60c5229e:::53:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:edcbc575:::54:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:102cf253:::55:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:efb7fb0b:::56:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:50d0a326:::57:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d4dc5daf:::58:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:3a130462:::59:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:ec87ed71:::60:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d5bc9454:::61:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:3ddfe313:::62:head 2024-09-18T11:49:24.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:7c2816b9:::63:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:47e00e4d:::64:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c6410c18:::65:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b48ed237:::66:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:cd63ad31:::67:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b179e92b:::68:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:0d9f741a:::69:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:6d3352ae:::70:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c6d5c19e:::71:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:bc4729c3:::72:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:77e930b9:::73:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:0abeecfd:::74:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b7c37e15:::75:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b6378398:::76:head 2024-09-18T11:49:24.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:02bd68de:::77:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:cc795d2d:::78:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:630d4fea:::79:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:e0d29ef5:::80:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:fd6f13d2:::81:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:606461d5:::82:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:eadbdc43:::83:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:8761d0bb:::84:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:9ef0186f:::85:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:e0d41294:::86:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:961de695:::87:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:1423148f:::88:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:633a8fa2:::89:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a8653809:::90:head 2024-09-18T11:49:24.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:3dac8b33:::91:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:35aad435:::92:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f6dcc343:::93:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:dbbdad87:::94:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:1cb48ce0:::95:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:03cd461c:::96:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:17a4ea99:::97:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:9993c9a7:::98:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:6394211c:::99:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:94c7ae57:::100:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:6fdee5bb:::101:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:9a477fd1:::102:head 2024-09-18T11:49:24.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:eb850916:::103:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:affc56b9:::104:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b42dc814:::105:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f319f8f0:::106:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:9a40b9de:::107:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:8b524f28:::108:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:e3de589f:::109:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:90f90a5b:::110:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a7b4f1d7:::111:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:af51766e:::112:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b6f90bd1:::113:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:e0261208:::114:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c9569ef7:::115:head 2024-09-18T11:49:24.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:61bebe50:::116:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:fe93412b:::117:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d3d38bee:::118:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:3100ba0c:::119:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d0560ada:::120:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f0ea8b35:::121:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:766f231a:::122:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a07a2582:::123:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:bd7c6b3a:::124:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:fb2ddaff:::125:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:4408e1fe:::126:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:ee1df7a7:::127:head 2024-09-18T11:49:24.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c3002909:::128:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:4f48ffa9:::129:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:edf38733:::130:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c08425c0:::131:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:5f902d98:::132:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:41ea2c93:::133:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:813cee13:::134:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:0131818d:::135:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:26ba5a85:::136:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:381b8a5a:::137:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:28797e47:::138:head 2024-09-18T11:49:24.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:bfca7f22:::139:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:36807075:::140:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:80b03975:::141:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:5c15709b:::142:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f39ea15e:::143:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:ea992956:::144:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:48887b1c:::145:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:9f24a9dd:::146:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:987f100b:::147:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d2dd3581:::148:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:7fed1808:::149:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c80b70e9:::150:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:85ed90f9:::151:head 2024-09-18T11:49:24.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:36428b24:::152:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d044c34a:::153:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:7c18bf58:::154:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d1c21232:::155:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a7a3c575:::156:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:87da0633:::157:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d5ac3822:::158:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:3f20522d:::159:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:6ca26563:::160:head 2024-09-18T11:49:24.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:532ce135:::161:head 2024-09-18T11:49:24.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:c78863e6:::162:head 2024-09-18T11:49:53.167 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T11:49:53.167 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T11:49:53.167 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T11:49:53.167 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T11:49:53.171 INFO:tasks.ceph.mon.a:Started 2024-09-18T11:49:53.171 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:49:53.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:49:53.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.273+0000 7f3ce6ef8700 10 monclient: get_monmap_and_config 2024-09-18T11:49:53.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.273+0000 7f3ce6ef8700 10 monclient: build_initial_monmap 2024-09-18T11:49:53.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient: monmap: 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:53.280385+0000 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:53.280385+0000 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:53.281 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 1 Processor -- start 2024-09-18T11:49:53.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 1 -- start start 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient: init 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient(hunting): picked mon.noname-b con 0x7f3ce00a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient(hunting): picked mon.noname-a con 0x7f3ce00a0e60 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient(hunting): picked mon.noname-c con 0x7f3ce0097970 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce0097550 con 0x7f3ce00a0e60 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce0096cb0 con 0x7f3ce0097970 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce00a1440 con 0x7f3ce00a0380 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce6ef8700 10 monclient(hunting): authenticate will time out at 1165.121956s 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce5ef6700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3ce00a0380 0x7f3ce00a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59728/0 (socket says 172.21.15.112:59728) 2024-09-18T11:49:53.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.277+0000 7f3ce5ef6700 1 -- 172.21.15.112:0/3982297099 learned_addr learned my addr 172.21.15.112:0/3982297099 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:53.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce56f5700 1 -- 172.21.15.112:0/3982297099 >> v1:172.21.15.112:6789/0 conn(0x7f3ce00a0e60 legacy=0x7f3ce0137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce5ef6700 1 -- 172.21.15.112:0/3982297099 >> v1:172.21.15.112:6790/0 conn(0x7f3ce00a0380 legacy=0x7f3ce00a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce66f7700 1 -- 172.21.15.112:0/3982297099 >> v1:172.21.15.129:6789/0 conn(0x7f3ce0097970 legacy=0x7f3ce013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782178192 0 0) 0x7f3ce0096cb0 con 0x7f3ce0097970 2024-09-18T11:49:53.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient(hunting): my global_id is 18025 2024-09-18T11:49:53.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cc8003570 con 0x7f3ce0097970 2024-09-18T11:49:53.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733230860 0 0) 0x7f3ce00a1440 con 0x7f3ce00a0380 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient(hunting): my global_id is 84302 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce0096cb0 con 0x7f3ce00a0380 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1673403864 0 0) 0x7f3cc8003570 con 0x7f3ce0097970 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 >> v1:172.21.15.112:6790/0 conn(0x7f3ce00a0380 legacy=0x7f3ce00a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 >> v1:172.21.15.112:6789/0 conn(0x7f3ce00a0e60 legacy=0x7f3ce0137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: _finish_hunting 0 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 monclient: found mon.noname-c 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:49:53.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce00a1de0 con 0x7f3ce0097970 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: _finish_auth 0 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3cd8001990 con 0x7f3ce0097970 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: dump: 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.289 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 5 monclient: authenticate success, global_id 18025 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cd8001c20 con 0x7f3ce0097970 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 1 -- 172.21.15.112:0/3982297099 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3cd80020a0 con 0x7f3ce0097970 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 4 set_mon_vals no callback set 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce4ef4700 10 monclient: dump: 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.290 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 10 monclient: shutdown 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/3982297099 >> v1:172.21.15.129:6789/0 conn(0x7f3ce0097970 legacy=0x7f3ce013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/3982297099 shutdown_connections 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/3982297099 >> 172.21.15.112:0/3982297099 conn(0x7f3ce0068100 msgr2=0x7f3ce00a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:53.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/3982297099 shutdown_connections 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/3982297099 wait complete. 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 10 monclient: build_initial_monmap 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 10 monclient: monmap: 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:53.292 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 librados: starting msgr at 2024-09-18T11:49:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 librados: starting objecter 2024-09-18T11:49:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 Processor -- start 2024-09-18T11:49:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 -- start start 2024-09-18T11:49:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 librados: setting wanted keys 2024-09-18T11:49:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 1 librados: calling monclient init 2024-09-18T11:49:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.281+0000 7f3ce6ef8700 10 monclient: init 2024-09-18T11:49:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 10 monclient(hunting): picked mon.noname-a con 0x7f3ce0097970 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 10 monclient(hunting): picked mon.noname-c con 0x7f3ce00a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 10 monclient(hunting): picked mon.noname-b con 0x7f3ce00a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce00a2050 con 0x7f3ce0097970 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce0134e40 con 0x7f3ce00a0e60 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce013aee0 con 0x7f3ce00a0380 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce6ef8700 10 monclient(hunting): authenticate will time out at 1165.128346s 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce5ef6700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3ce0097970 0x7f3ce01416a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:53972/0 (socket says 172.21.15.112:53972) 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce5ef6700 1 -- 172.21.15.112:0/594861304 learned_addr learned my addr 172.21.15.112:0/594861304 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce5ef6700 1 -- 172.21.15.112:0/594861304 >> v1:172.21.15.112:6789/0 conn(0x7f3ce0097970 legacy=0x7f3ce01416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 client.?.objecter ms_handle_connect 0x7f3ce0097970 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 client.?.objecter ms_handle_connect 0x7f3ce00a0380 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce56f5700 1 -- 172.21.15.112:0/594861304 >> v1:172.21.15.112:6790/0 conn(0x7f3ce00a0380 legacy=0x7f3ce0134730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809792947 0 0) 0x7f3ce013aee0 con 0x7f3ce00a0380 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): my global_id is 84305 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ccc004f10 con 0x7f3ce00a0380 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 client.?.objecter ms_handle_connect 0x7f3ce00a0e60 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce66f7700 1 -- 172.21.15.112:0/594861304 >> v1:172.21.15.129:6789/0 conn(0x7f3ce00a0e60 legacy=0x7f3ce0149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3ce56f5700 1 --1- 172.21.15.112:0/594861304 >> v1:172.21.15.112:6790/0 conn(0x7f3ce00a0380 0x7f3ce0134730 :-1 s=OPENED pgs=8923 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f3ccc004f10 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4207708626 0 0) 0x7f3ccc004f10 con 0x7f3ce00a0380 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ce013aee0 con 0x7f3ce00a0380 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190018608 0 0) 0x7f3ce0134e40 con 0x7f3ce00a0e60 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): my global_id is 18028 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ccc004f10 con 0x7f3ce00a0e60 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3cd4001bc0 con 0x7f3ce00a0380 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): dump: 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.300 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.285+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 207790215 0 0) 0x7f3ccc004f10 con 0x7f3ce00a0e60 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ce0134e40 con 0x7f3ce00a0e60 2024-09-18T11:49:53.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3cd80020a0 con 0x7f3ce00a0e60 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient(hunting): dump: 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.302 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1420160751 0 0) 0x7f3ce0134e40 con 0x7f3ce00a0e60 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 >> v1:172.21.15.112:6790/0 conn(0x7f3ce00a0380 legacy=0x7f3ce0134730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 >> v1:172.21.15.112:6789/0 conn(0x7f3ce0097970 legacy=0x7f3ce01416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient: _finish_hunting 0 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 monclient: found mon.b 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce01395e0 con 0x7f3ce00a0e60 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient: _finish_auth 0 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 5 monclient: authenticate success, global_id 18028 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 10 monclient: _renew_subs 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:49:53.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce013ac60 con 0x7f3ce00a0e60 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2264220033 0 0) 0x7f3ce013aee0 con 0x7f3ce00a0380 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 10 client.18028.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 10 monclient: _renew_subs 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce0139ca0 con 0x7f3ce00a0e60 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 1 librados: init done 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cd80015d0 con 0x7f3ce00a0e60 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3cd80027f0 con 0x7f3ce00a0e60 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:49:53.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cdffff700 4 set_mon_vals no callback set 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 monclient: dump: 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:53.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc00050b0 con 0x7f3ce00a0e60 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f3cd8002a10 con 0x7f3ce00a0e60 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(257..257 src has 1..257) v4 ==== 6105+0+0 (unknown 192619946 0 0) 0x7f3cd8044130 con 0x7f3ce00a0e60 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 10 client.18028.objecter ms_dispatch 0x7f3ce0000fd0 osd_map(257..257 src has 1..257) v4 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 3 client.18028.objecter handle_osd_map got epochs [257,257] > 0 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 3 client.18028.objecter handle_osd_map decoding full epoch 257 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3cddffb700 20 client.18028.objecter dump_active .. 0 homeless 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.289+0000 7f3ce5ef6700 10 monclient: get_auth_request con 0x7f3ccc038290 auth_method 0 2024-09-18T11:49:53.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.293+0000 7f3cddffb700 10 client.18028.objecter ms_handle_connect 0x7f3ccc038290 2024-09-18T11:49:53.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.301+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f3cd8014a90 con 0x7f3ce00a0e60 2024-09-18T11:49:53.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.301+0000 7f3cddffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:53.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.301+0000 7f3cddffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:53.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.537+0000 7f3ce6ef8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:49:53.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.537+0000 7f3ce6ef8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:49:53.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.537+0000 7f3ce6ef8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:49:53.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.537+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3cc00058e0 con 0x7f3ce00a0e60 2024-09-18T11:49:53.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.537+0000 7f3cddffb700 1 -- 172.21.15.112:0/594861304 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1370294094) 0x7f3cd8014490 con 0x7f3ce00a0e60 2024-09-18T11:49:53.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3cddffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:49:53.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3cddffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:53.544 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:49:53.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 10 librados: watch_flush enter 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 10 librados: watch_flush exit 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 20 client.18028.objecter shutdown clearing up homeless session... 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 10 client.18028.objecter successfully canceled tick 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 >> v1:172.21.15.129:6812/20806 conn(0x7f3ccc038290 legacy=0x7f3ccc03a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 10 monclient: shutdown 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 >> v1:172.21.15.129:6789/0 conn(0x7f3ce00a0e60 legacy=0x7f3ce0149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 shutdown_connections 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 >> 172.21.15.112:0/594861304 conn(0x7f3ce0068100 msgr2=0x7f3ce013b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 shutdown_connections 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 1 -- 172.21.15.112:0/594861304 wait complete. 2024-09-18T11:49:53.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.541+0000 7f3ce6ef8700 1 librados: shutdown 2024-09-18T11:49:53.558 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:49:53.558 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:49:53.558 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:49:53.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient: get_monmap_and_config 2024-09-18T11:49:53.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient: build_initial_monmap 2024-09-18T11:49:53.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient: monmap: 2024-09-18T11:49:53.670 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:53.669989+0000 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:53.669989+0000 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 1 Processor -- start 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 1 -- start start 2024-09-18T11:49:53.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient: init 2024-09-18T11:49:53.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:53.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:49:53.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f38100a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f3810096a60 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f381013ca20 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381006e910 con 0x7f38100a1ea0 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381006e630 con 0x7f381013ca20 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38100a2270 con 0x7f3810096a60 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380f59e700 10 monclient(hunting): authenticate will time out at 1165.511028s 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380dd9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3810096a60 0x7f381006c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59748/0 (socket says 172.21.15.112:59748) 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380dd9b700 1 -- 172.21.15.112:0/106987437 learned_addr learned my addr 172.21.15.112:0/106987437 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:53.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.665+0000 7f380e59c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f38100a1ea0 0x7f381006bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:53976/0 (socket says 172.21.15.112:53976) 2024-09-18T11:49:53.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.669+0000 7f380dd9b700 1 -- 172.21.15.112:0/106987437 >> v1:172.21.15.112:6790/0 conn(0x7f3810096a60 legacy=0x7f381006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.669+0000 7f380e59c700 1 -- 172.21.15.112:0/106987437 >> v1:172.21.15.112:6789/0 conn(0x7f38100a1ea0 legacy=0x7f381006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.669+0000 7f380ed9d700 1 -- 172.21.15.112:0/106987437 >> v1:172.21.15.129:6789/0 conn(0x7f381013ca20 legacy=0x7f381013ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.669+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557405339 0 0) 0x7f381006e910 con 0x7f38100a1ea0 2024-09-18T11:49:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.669+0000 7f380d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.669+0000 7f380d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.669+0000 7f380d59a700 10 monclient(hunting): my global_id is 94098 2024-09-18T11:49:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.669+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f8003570 con 0x7f38100a1ea0 2024-09-18T11:49:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034416704 0 0) 0x7f37f8003570 con 0x7f38100a1ea0 2024-09-18T11:49:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 >> v1:172.21.15.112:6790/0 conn(0x7f3810096a60 legacy=0x7f381006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 >> v1:172.21.15.129:6789/0 conn(0x7f381013ca20 legacy=0x7f381013ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: _finish_hunting 0 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 1 monclient: found mon.noname-a 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381006eed0 con 0x7f38100a1ea0 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: _finish_auth 0 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 5 monclient: authenticate success, global_id 94098 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3804001940 con 0x7f38100a1ea0 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: dump: 2024-09-18T11:49:53.680 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3804001c10 con 0x7f38100a1ea0 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:53.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 4 set_mon_vals no callback set 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 1 -- 172.21.15.112:0/106987437 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f38040049d0 con 0x7f38100a1ea0 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380d59a700 10 monclient: dump: 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.682 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient: shutdown 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- 172.21.15.112:0/106987437 >> v1:172.21.15.112:6789/0 conn(0x7f38100a1ea0 legacy=0x7f381006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- 172.21.15.112:0/106987437 shutdown_connections 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- 172.21.15.112:0/106987437 >> 172.21.15.112:0/106987437 conn(0x7f3810067da0 msgr2=0x7f38100a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- 172.21.15.112:0/106987437 shutdown_connections 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- 172.21.15.112:0/106987437 wait complete. 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient: build_initial_monmap 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient: monmap: 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:53.683 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 librados: starting msgr at 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 librados: starting objecter 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 Processor -- start 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- start start 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 librados: setting wanted keys 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 librados: calling monclient init 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient: init 2024-09-18T11:49:53.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f3810096a60 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f38100a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f381013ca20 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381006e110 con 0x7f381013ca20 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381009de70 con 0x7f38100a1ea0 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381013c7f0 con 0x7f3810096a60 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380f59e700 10 monclient(hunting): authenticate will time out at 1165.518734s 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380ed9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f381013ca20 0x7f381014ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:53982/0 (socket says 172.21.15.112:53982) 2024-09-18T11:49:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.673+0000 7f380ed9d700 1 -- 172.21.15.112:0/2157701204 learned_addr learned my addr 172.21.15.112:0/2157701204 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380ed9d700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6789/0 conn(0x7f381013ca20 legacy=0x7f381014ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 client.?.objecter ms_handle_connect 0x7f381013ca20 2024-09-18T11:49:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 client.?.objecter ms_handle_connect 0x7f3810096a60 2024-09-18T11:49:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380e59c700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6790/0 conn(0x7f3810096a60 legacy=0x7f38101233c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357431146 0 0) 0x7f381006e110 con 0x7f381013ca20 2024-09-18T11:49:53.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient(hunting): my global_id is 94101 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f0004f10 con 0x7f381013ca20 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 client.?.objecter ms_handle_connect 0x7f38100a1ea0 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380dd9b700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.129:6789/0 conn(0x7f38100a1ea0 legacy=0x7f3810123ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090185258 0 0) 0x7f37f0004f10 con 0x7f381013ca20 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f381006e110 con 0x7f381013ca20 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3808001fc0 con 0x7f381013ca20 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient(hunting): dump: 2024-09-18T11:49:53.691 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1609004885 0 0) 0x7f381006e110 con 0x7f381013ca20 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6790/0 conn(0x7f3810096a60 legacy=0x7f38101233c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.129:6789/0 conn(0x7f38100a1ea0 legacy=0x7f3810123ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 monclient: found mon.a 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3810124c40 con 0x7f381013ca20 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient: _finish_auth 0 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 5 monclient: authenticate success, global_id 94101 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 10 monclient: _renew_subs 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3810126770 con 0x7f381013ca20 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 10 client.94101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:53.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 10 monclient: _renew_subs 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38101259b0 con 0x7f381013ca20 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 1 librados: init done 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3808002630 con 0x7f381013ca20 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3808002980 con 0x7f381013ca20 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380cd99700 4 set_mon_vals no callback set 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 10 monclient: dump: 2024-09-18T11:49:53.694 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f37f4006590 con 0x7f37f4002840 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.677+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f3808002ae0 con 0x7f381013ca20 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(258..258 src has 1..258) v4 ==== 6105+0+0 (unknown 349540209 0 0) 0x7f3808044590 con 0x7f381013ca20 2024-09-18T11:49:53.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 client.94101.objecter ms_dispatch 0x7f3810000fd0 osd_map(258..258 src has 1..258) v4 2024-09-18T11:49:53.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 3 client.94101.objecter handle_osd_map got epochs [258,258] > 0 2024-09-18T11:49:53.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 3 client.94101.objecter handle_osd_map decoding full epoch 258 2024-09-18T11:49:53.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 20 client.94101.objecter dump_active .. 0 homeless 2024-09-18T11:49:53.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f380dd9b700 10 monclient: get_auth_request con 0x7f37f0038110 auth_method 0 2024-09-18T11:49:53.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 client.94101.objecter ms_handle_connect 0x7f37f4002840 2024-09-18T11:49:53.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 client.94101.objecter resend_mon_ops 2024-09-18T11:49:53.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f380e59c700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6789/0 conn(0x7f37f4002840 legacy=0x7f37f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822548530 0 0) 0x7f37f4006590 con 0x7f37f4002840 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 monclient(hunting): my global_id is 94104 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f381006e110 con 0x7f37f4002840 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2061775240 0 0) 0x7f381006e110 con 0x7f37f4002840 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f4006590 con 0x7f37f4002840 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3804004aa0 con 0x7f37f4002840 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:53.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 monclient: dump: 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 746841004 0 0) 0x7f37f4006590 con 0x7f37f4002840 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f400a140 con 0x7f37f4002840 2024-09-18T11:49:53.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 client.94101.objecter ms_handle_connect 0x7f37f0038110 2024-09-18T11:49:53.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f38040017a0 con 0x7f37f4002840 2024-09-18T11:49:53.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:53.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:53.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6789/0 conn(0x7f37f4002840 legacy=0x7f37f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.681+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6789/0 conn(0x7f37f4002840 legacy=0x7f37f4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:53.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f380f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:49:53.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f37f4009670 con 0x7f37f40090e0 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f380e59c700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6789/0 conn(0x7f37f40090e0 legacy=0x7f37f4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 10 client.94101.objecter ms_handle_connect 0x7f37f40090e0 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 10 client.94101.objecter resend_mon_ops 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874901383 0 0) 0x7f37f4009670 con 0x7f37f40090e0 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 10 monclient(hunting): my global_id is 94107 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f4006590 con 0x7f37f40090e0 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1572223460 0 0) 0x7f37f4006590 con 0x7f37f40090e0 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f4009670 con 0x7f37f40090e0 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3804005390 con 0x7f37f40090e0 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:53.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 10 monclient: dump: 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4054007191 0 0) 0x7f37f4009670 con 0x7f37f40090e0 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.693+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f37f4009930 con 0x7f37f40090e0 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 1918804819) 0x7f3804006f10 con 0x7f37f40090e0 2024-09-18T11:49:53.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f37fe7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:49:53.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f37fe7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:53.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6789/0 conn(0x7f37f40090e0 legacy=0x7f37f4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f37fe7fc700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6789/0 conn(0x7f37f40090e0 legacy=0x7f37f4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:53.703 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:49:53.703 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:49:53.703 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:49:53.703 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 28, 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:53.704 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:49:53.705 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:49:53.706 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:53.706 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:53.706 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:53.706 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:53.706 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:53.706 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:53.707 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:53.707 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:53.707 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:53.707 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:53.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.707 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:49:53.708 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:49:53.708 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:49:53.708 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:49:53.708 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:49:53.708 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:49:53.708 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:53.708 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:53.709 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:49:53.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:53.711 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:53.712 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:53.713 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:49:53.714 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:53.715 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:49:53.716 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:49:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 10 librados: watch_flush enter 2024-09-18T11:49:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 10 librados: watch_flush exit 2024-09-18T11:49:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 20 client.94101.objecter shutdown clearing up homeless session... 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 10 client.94101.objecter successfully canceled tick 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.129:6812/20806 conn(0x7f37f0038110 legacy=0x7f37f003a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 10 monclient: shutdown 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 >> v1:172.21.15.112:6789/0 conn(0x7f381013ca20 legacy=0x7f381014ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380ed9d700 1 -- 172.21.15.112:0/2157701204 reap_dead start 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380ed9d700 1 -- 172.21.15.112:0/2157701204 reap_dead start 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 shutdown_connections 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 >> 172.21.15.112:0/2157701204 conn(0x7f3810067da0 msgr2=0x7f381009fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 shutdown_connections 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 1 -- 172.21.15.112:0/2157701204 wait complete. 2024-09-18T11:49:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.697+0000 7f380f59e700 1 librados: shutdown 2024-09-18T11:49:53.719 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:49:53.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:49:53.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:49:53.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient: get_monmap_and_config 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient: build_initial_monmap 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient: monmap: 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:53.833731+0000 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:53.833731+0000 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:53.834 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 1 Processor -- start 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 1 -- start start 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient: init 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient(hunting): picked mon.noname-a con 0x7fddc80a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient(hunting): picked mon.noname-b con 0x7fddc80a0750 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient(hunting): picked mon.noname-c con 0x7fddc81385b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc8097270 con 0x7fddc80a0380 2024-09-18T11:49:53.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc80a3170 con 0x7fddc81385b0 2024-09-18T11:49:53.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc8096cb0 con 0x7fddc80a0750 2024-09-18T11:49:53.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:53.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddce182700 10 monclient(hunting): authenticate will time out at 1165.675436s 2024-09-18T11:49:53.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddcd180700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fddc80a0380 0x7fddc8097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54002/0 (socket says 172.21.15.112:54002) 2024-09-18T11:49:53.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddcd180700 1 -- 172.21.15.112:0/1817794554 learned_addr learned my addr 172.21.15.112:0/1817794554 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:53.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.829+0000 7fddcc97f700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fddc80a0750 0x7fddc80a0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59770/0 (socket says 172.21.15.112:59770) 2024-09-18T11:49:53.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddcc97f700 1 -- 172.21.15.112:0/1817794554 >> v1:172.21.15.112:6790/0 conn(0x7fddc80a0750 legacy=0x7fddc80a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddcd180700 1 -- 172.21.15.112:0/1817794554 >> v1:172.21.15.112:6789/0 conn(0x7fddc80a0380 legacy=0x7fddc8097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddcd981700 1 -- 172.21.15.112:0/1817794554 >> v1:172.21.15.129:6789/0 conn(0x7fddc81385b0 legacy=0x7fddc813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607673795 0 0) 0x7fddc8097270 con 0x7fddc80a0380 2024-09-18T11:49:53.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient(hunting): my global_id is 94110 2024-09-18T11:49:53.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb4003570 con 0x7fddc80a0380 2024-09-18T11:49:53.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2404551113 0 0) 0x7fddb4003570 con 0x7fddc80a0380 2024-09-18T11:49:53.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 >> v1:172.21.15.112:6790/0 conn(0x7fddc80a0750 legacy=0x7fddc80a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 >> v1:172.21.15.129:6789/0 conn(0x7fddc81385b0 legacy=0x7fddc813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: _finish_hunting 0 2024-09-18T11:49:53.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 monclient: found mon.noname-a 2024-09-18T11:49:53.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:53.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:49:53.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc80a1420 con 0x7fddc80a0380 2024-09-18T11:49:53.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: _finish_auth 0 2024-09-18T11:49:53.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:53.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fddc4001a00 con 0x7fddc80a0380 2024-09-18T11:49:53.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddce182700 5 monclient: authenticate success, global_id 94110 2024-09-18T11:49:53.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddce182700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:53.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:53.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: dump: 2024-09-18T11:49:53.843 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.843 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddce182700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:53.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddc4001be0 con 0x7fddc80a0380 2024-09-18T11:49:53.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:53.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddce182700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:53.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddce182700 4 set_mon_vals no callback set 2024-09-18T11:49:53.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 1 -- 172.21.15.112:0/1817794554 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fddc4004a90 con 0x7fddc80a0380 2024-09-18T11:49:53.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:53.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.833+0000 7fddbffff700 10 monclient: dump: 2024-09-18T11:49:53.845 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.846 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient: shutdown 2024-09-18T11:49:53.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:53.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- 172.21.15.112:0/1817794554 >> v1:172.21.15.112:6789/0 conn(0x7fddc80a0380 legacy=0x7fddc8097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:53.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- 172.21.15.112:0/1817794554 shutdown_connections 2024-09-18T11:49:53.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- 172.21.15.112:0/1817794554 >> 172.21.15.112:0/1817794554 conn(0x7fddc80a5990 msgr2=0x7fddc80a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- 172.21.15.112:0/1817794554 shutdown_connections 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- 172.21.15.112:0/1817794554 wait complete. 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient: build_initial_monmap 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient: monmap: 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:53.850 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 librados: starting msgr at 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 librados: starting objecter 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 Processor -- start 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- start start 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 librados: setting wanted keys 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 librados: calling monclient init 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient: init 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient(hunting): picked mon.noname-c con 0x7fddc80a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient(hunting): picked mon.noname-a con 0x7fddc80a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient(hunting): picked mon.noname-b con 0x7fddc81385b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc8142450 con 0x7fddc80a0750 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc81377f0 con 0x7fddc81385b0 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc8138980 con 0x7fddc80a0380 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddce182700 10 monclient(hunting): authenticate will time out at 1165.683227s 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddcd180700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fddc80a0380 0x7fddc8141590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59784/0 (socket says 172.21.15.112:59784) 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.837+0000 7fddcd180700 1 -- 172.21.15.112:0/2369734258 learned_addr learned my addr 172.21.15.112:0/2369734258 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddcc97f700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6789/0 conn(0x7fddc80a0750 legacy=0x7fddc809e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddcd180700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6790/0 conn(0x7fddc80a0380 legacy=0x7fddc8141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 client.?.objecter ms_handle_connect 0x7fddc80a0750 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 client.?.objecter ms_handle_connect 0x7fddc80a0380 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddcd981700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.129:6789/0 conn(0x7fddc81385b0 legacy=0x7fddc81498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307870541 0 0) 0x7fddc8142450 con 0x7fddc80a0750 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 monclient(hunting): my global_id is 94113 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac004f10 con 0x7fddc80a0750 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 client.?.objecter ms_handle_connect 0x7fddc81385b0 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddcc97f700 1 --1- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6789/0 conn(0x7fddc80a0750 0x7fddc809e000 :-1 s=OPENED pgs=358 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7fddb8001b70 mon_map magic: 0 v1 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2969977331 0 0) 0x7fddac004f10 con 0x7fddc80a0750 2024-09-18T11:49:53.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddc8142450 con 0x7fddc80a0750 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fddb8001b70 con 0x7fddc80a0750 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 monclient(hunting): dump: 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:53.854 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:53.855 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:53.855 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:53.855 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:53.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:53.841+0000 7fddbd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:54.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726020797 0 0) 0x7fddc8138980 con 0x7fddc80a0380 2024-09-18T11:49:54.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 10 monclient(hunting): my global_id is 84308 2024-09-18T11:49:54.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac004f10 con 0x7fddc80a0380 2024-09-18T11:49:54.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876452985 0 0) 0x7fddc81377f0 con 0x7fddc81385b0 2024-09-18T11:49:54.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 10 monclient(hunting): my global_id is 18031 2024-09-18T11:49:54.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.173+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc8138980 con 0x7fddc81385b0 2024-09-18T11:49:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4232980162 0 0) 0x7fddac004f10 con 0x7fddc80a0380 2024-09-18T11:49:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddc81377f0 con 0x7fddc80a0380 2024-09-18T11:49:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fddc4001d90 con 0x7fddc80a0380 2024-09-18T11:49:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient(hunting): dump: 2024-09-18T11:49:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2440818744 0 0) 0x7fddc81377f0 con 0x7fddc80a0380 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.129:6789/0 conn(0x7fddc81385b0 legacy=0x7fddc81498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6789/0 conn(0x7fddc80a0750 legacy=0x7fddc809e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient: _finish_hunting 0 2024-09-18T11:49:54.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 monclient: found mon.c 2024-09-18T11:49:54.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:54.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:54.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc80a2010 con 0x7fddc80a0380 2024-09-18T11:49:54.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient: _finish_auth 0 2024-09-18T11:49:54.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:54.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 355651616 0 0) 0x7fddc8142450 con 0x7fddc80a0750 2024-09-18T11:49:54.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 5 monclient: authenticate success, global_id 84308 2024-09-18T11:49:54.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 10 monclient: _renew_subs 2024-09-18T11:49:54.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:54.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc8137e80 con 0x7fddc80a0380 2024-09-18T11:49:54.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:49:54.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 10 client.84308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:54.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 10 monclient: _renew_subs 2024-09-18T11:49:54.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:54.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc809e710 con 0x7fddc80a0380 2024-09-18T11:49:54.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 1 librados: init done 2024-09-18T11:49:54.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddc40049f0 con 0x7fddc80a0380 2024-09-18T11:49:54.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:54.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fddc4005290 con 0x7fddc80a0380 2024-09-18T11:49:54.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:54.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbf7fe700 4 set_mon_vals no callback set 2024-09-18T11:49:54.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddbd7fa700 10 monclient: dump: 2024-09-18T11:49:54.196 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.197 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.197 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.197 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.197 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.197 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.197 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.198 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.198 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.198 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.185+0000 7fddce182700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:54.202 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdda8006590 con 0x7fdda8002840 2024-09-18T11:49:54.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fddc400fb10 con 0x7fddc80a0380 2024-09-18T11:49:54.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(259..259 src has 1..259) v4 ==== 5676+0+0 (unknown 340334306 0 0) 0x7fddc4044070 con 0x7fddc80a0380 2024-09-18T11:49:54.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 client.84308.objecter ms_dispatch 0x7fddc8000fd0 osd_map(259..259 src has 1..259) v4 2024-09-18T11:49:54.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 3 client.84308.objecter handle_osd_map got epochs [259,259] > 0 2024-09-18T11:49:54.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 3 client.84308.objecter handle_osd_map decoding full epoch 259 2024-09-18T11:49:54.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 20 client.84308.objecter dump_active .. 0 homeless 2024-09-18T11:49:54.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddcd981700 10 monclient: get_auth_request con 0x7fddac0385e0 auth_method 0 2024-09-18T11:49:54.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddcc97f700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6790/0 conn(0x7fdda8002840 legacy=0x7fdda8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 client.84308.objecter ms_handle_connect 0x7fdda8002840 2024-09-18T11:49:54.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 client.84308.objecter resend_mon_ops 2024-09-18T11:49:54.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426973706 0 0) 0x7fdda8006590 con 0x7fdda8002840 2024-09-18T11:49:54.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 monclient(hunting): my global_id is 84311 2024-09-18T11:49:54.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc8142450 con 0x7fdda8002840 2024-09-18T11:49:54.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 153311678 0 0) 0x7fddc8142450 con 0x7fdda8002840 2024-09-18T11:49:54.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdda8006590 con 0x7fdda8002840 2024-09-18T11:49:54.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fddb8005070 con 0x7fdda8002840 2024-09-18T11:49:54.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:54.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 monclient: dump: 2024-09-18T11:49:54.206 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.206 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.207 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.207 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.207 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.207 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.207 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.207 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.208 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.208 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 324797676 0 0) 0x7fdda8006590 con 0x7fdda8002840 2024-09-18T11:49:54.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdda800a140 con 0x7fdda8002840 2024-09-18T11:49:54.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.189+0000 7fddbd7fa700 10 client.84308.objecter ms_handle_connect 0x7fddac0385e0 2024-09-18T11:49:54.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.193+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fddb80051d0 con 0x7fdda8002840 2024-09-18T11:49:54.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.193+0000 7fddbd7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:54.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.193+0000 7fddbd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:54.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.193+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6790/0 conn(0x7fdda8002840 legacy=0x7fdda8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.193+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6790/0 conn(0x7fdda8002840 legacy=0x7fdda8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:54.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.201+0000 7fddce182700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:49:54.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.201+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdda8009650 con 0x7fdda80090c0 2024-09-18T11:49:54.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddcc97f700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6790/0 conn(0x7fdda80090c0 legacy=0x7fdda8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 client.84308.objecter ms_handle_connect 0x7fdda80090c0 2024-09-18T11:49:54.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 client.84308.objecter resend_mon_ops 2024-09-18T11:49:54.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458125985 0 0) 0x7fdda8009650 con 0x7fdda80090c0 2024-09-18T11:49:54.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 monclient(hunting): my global_id is 84314 2024-09-18T11:49:54.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8006590 con 0x7fdda80090c0 2024-09-18T11:49:54.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 212261557 0 0) 0x7fdda8006590 con 0x7fdda80090c0 2024-09-18T11:49:54.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdda8009650 con 0x7fdda80090c0 2024-09-18T11:49:54.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fddb8005a00 con 0x7fdda80090c0 2024-09-18T11:49:54.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:54.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 monclient: dump: 2024-09-18T11:49:54.215 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.215 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.216 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.216 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.216 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.216 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.216 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.216 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.217 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.217 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1386669018 0 0) 0x7fdda8009650 con 0x7fdda80090c0 2024-09-18T11:49:54.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdda8009910 con 0x7fdda80090c0 2024-09-18T11:49:54.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 1056412564) 0x7fddb8005b60 con 0x7fdda80090c0 2024-09-18T11:49:54.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:49:54.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:54.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6790/0 conn(0x7fdda80090c0 legacy=0x7fdda8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddbd7fa700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6790/0 conn(0x7fdda80090c0 legacy=0x7fdda8008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:54.219 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:49:54.219 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:49:54.219 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:49:54.219 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:49:54.219 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 28, 2024-09-18T11:49:54.219 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:49:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:49:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:49:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:49:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:49:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:49:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:49:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:49:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:49:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:49:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:49:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:49:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:49:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:49:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:49:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:49:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:49:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:49:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:49:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:49:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:49:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:49:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:49:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:49:54.229 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.230 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:54.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:54.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:54.233 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:49:54.234 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:49:54.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 10 librados: watch_flush enter 2024-09-18T11:49:54.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 10 librados: watch_flush exit 2024-09-18T11:49:54.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 20 client.84308.objecter shutdown clearing up homeless session... 2024-09-18T11:49:54.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 10 client.84308.objecter successfully canceled tick 2024-09-18T11:49:54.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.129:6812/20806 conn(0x7fddac0385e0 legacy=0x7fddac03aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 10 monclient: shutdown 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 >> v1:172.21.15.112:6790/0 conn(0x7fddc80a0380 legacy=0x7fddc8141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddcd981700 1 -- 172.21.15.112:0/2369734258 reap_dead start 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddcd981700 1 -- 172.21.15.112:0/2369734258 reap_dead start 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 shutdown_connections 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 >> 172.21.15.112:0/2369734258 conn(0x7fddc80a5990 msgr2=0x7fddc8138b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 shutdown_connections 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 1 -- 172.21.15.112:0/2369734258 wait complete. 2024-09-18T11:49:54.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.205+0000 7fddce182700 1 librados: shutdown 2024-09-18T11:49:54.238 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:49:54.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:49:54.238 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:49:54.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient: get_monmap_and_config 2024-09-18T11:49:54.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient: build_initial_monmap 2024-09-18T11:49:54.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient: monmap: 2024-09-18T11:49:54.350 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:54.350 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.350 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:54.349686+0000 2024-09-18T11:49:54.350 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:54.349686+0000 2024-09-18T11:49:54.350 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:54.350 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.351 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:54.351 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:54.351 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:54.351 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 1 Processor -- start 2024-09-18T11:49:54.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 1 -- start start 2024-09-18T11:49:54.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient: init 2024-09-18T11:49:54.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient(hunting): picked mon.noname-b con 0x7fc9e00a0570 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient(hunting): picked mon.noname-a con 0x7fc9e00a1050 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient(hunting): picked mon.noname-c con 0x7fc9e0097a50 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e0097070 con 0x7fc9e00a1050 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e0096d90 con 0x7fc9e0097a50 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e0097e50 con 0x7fc9e00a0570 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9e6416700 10 monclient(hunting): authenticate will time out at 1166.191537s 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9df7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc9e00a1050 0x7fc9e0137780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54032/0 (socket says 172.21.15.112:54032) 2024-09-18T11:49:54.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9df7fe700 1 -- 172.21.15.112:0/2932228444 learned_addr learned my addr 172.21.15.112:0/2932228444 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:54.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.345+0000 7fc9dffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc9e00a0570 0x7fc9e00a0940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59826/0 (socket says 172.21.15.112:59826) 2024-09-18T11:49:54.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9dffff700 1 -- 172.21.15.112:0/2932228444 >> v1:172.21.15.112:6790/0 conn(0x7fc9e00a0570 legacy=0x7fc9e00a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9df7fe700 1 -- 172.21.15.112:0/2932228444 >> v1:172.21.15.112:6789/0 conn(0x7fc9e00a1050 legacy=0x7fc9e0137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449362585 0 0) 0x7fc9e0097070 con 0x7fc9e00a1050 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): my global_id is 94116 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c4003570 con 0x7fc9e00a1050 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319536991 0 0) 0x7fc9e0097e50 con 0x7fc9e00a0570 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9e49b3700 1 -- 172.21.15.112:0/2932228444 >> v1:172.21.15.129:6789/0 conn(0x7fc9e0097a50 legacy=0x7fc9e013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): my global_id is 84317 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9df7fe700 1 -- 172.21.15.112:0/2932228444 >> v1:172.21.15.112:6789/0 conn(0x7fc9e00a1050 legacy=0x7fc9e0137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9df7fe700 1 -- 172.21.15.112:0/2932228444 >> v1:172.21.15.112:6789/0 conn(0x7fc9e00a1050 legacy=0x7fc9e0137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9df7fe700 1 --1- 172.21.15.112:0/2932228444 >> v1:172.21.15.112:6789/0 conn(0x7fc9e00a1050 0x7fc9e0137780 :-1 s=OPENED pgs=361 cs=1 l=1).handle_message read tag failed 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9df7fe700 1 --1- 172.21.15.112:0/2932228444 >> v1:172.21.15.112:6789/0 conn(0x7fc9e00a1050 0x7fc9e0137780 :-1 s=OPENED pgs=361 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e0097070 con 0x7fc9e00a0570 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6789/0 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212382271 0 0) 0x7fc9e0096d90 con 0x7fc9e0097a50 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient(hunting): my global_id is 18034 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e0097e50 con 0x7fc9e0097a50 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 858640740 0 0) 0x7fc9e0097070 con 0x7fc9e00a0570 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 >> v1:172.21.15.129:6789/0 conn(0x7fc9e0097a50 legacy=0x7fc9e013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 >> v1:172.21.15.112:6789/0 conn(0x7fc9e00a1050 legacy=0x7fc9e0137780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: _finish_hunting 0 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 monclient: found mon.noname-b 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e00a2050 con 0x7fc9e00a0570 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: _finish_auth 0 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc9d0001920 con 0x7fc9e00a0570 2024-09-18T11:49:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9e6416700 5 monclient: authenticate success, global_id 84317 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9e6416700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: dump: 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.359 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9e6416700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d0001bb0 con 0x7fc9e00a0570 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 1 -- 172.21.15.112:0/2932228444 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc9d00049d0 con 0x7fc9e00a0570 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9e6416700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9e6416700 4 set_mon_vals no callback set 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.349+0000 7fc9deffd700 10 monclient: dump: 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.362 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.362 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient: shutdown 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- 172.21.15.112:0/2932228444 >> v1:172.21.15.112:6790/0 conn(0x7fc9e00a0570 legacy=0x7fc9e00a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- 172.21.15.112:0/2932228444 shutdown_connections 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- 172.21.15.112:0/2932228444 >> 172.21.15.112:0/2932228444 conn(0x7fc9e0067680 msgr2=0x7fc9e0067a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- 172.21.15.112:0/2932228444 shutdown_connections 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- 172.21.15.112:0/2932228444 wait complete. 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient: build_initial_monmap 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient: monmap: 2024-09-18T11:49:54.369 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 librados: starting msgr at 2024-09-18T11:49:54.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 librados: starting objecter 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 Processor -- start 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- start start 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 librados: setting wanted keys 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 librados: calling monclient init 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient: init 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient(hunting): picked mon.noname-a con 0x7fc9e0097a50 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient(hunting): picked mon.noname-c con 0x7fc9e00a0570 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient(hunting): picked mon.noname-b con 0x7fc9e00a1050 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e0122a00 con 0x7fc9e0097a50 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e013dd90 con 0x7fc9e00a1050 2024-09-18T11:49:54.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e0120490 con 0x7fc9e00a0570 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9e6416700 10 monclient(hunting): authenticate will time out at 1166.198631s 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9df7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc9e00a0570 0x7fc9e01468d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59834/0 (socket says 172.21.15.112:59834) 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.353+0000 7fc9df7fe700 1 -- 172.21.15.112:0/1383900774 learned_addr learned my addr 172.21.15.112:0/1383900774 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9df7fe700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.112:6790/0 conn(0x7fc9e00a0570 legacy=0x7fc9e01468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 client.?.objecter ms_handle_connect 0x7fc9e00a0570 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9dffff700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.112:6789/0 conn(0x7fc9e0097a50 legacy=0x7fc9e013d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 client.?.objecter ms_handle_connect 0x7fc9e0097a50 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e49b3700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6789/0 conn(0x7fc9e00a1050 legacy=0x7fc9e014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029471101 0 0) 0x7fc9e0120490 con 0x7fc9e00a0570 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): my global_id is 84320 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c8004f10 con 0x7fc9e00a0570 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 client.?.objecter ms_handle_connect 0x7fc9e00a1050 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531708644 0 0) 0x7fc9e0122a00 con 0x7fc9e0097a50 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): my global_id is 94119 2024-09-18T11:49:54.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e0120490 con 0x7fc9e0097a50 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611031133 0 0) 0x7fc9e013dd90 con 0x7fc9e00a1050 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): my global_id is 18037 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e0122a00 con 0x7fc9e00a1050 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2918947040 0 0) 0x7fc9e0120490 con 0x7fc9e0097a50 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9e013dd90 con 0x7fc9e0097a50 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc9d0004830 con 0x7fc9e0097a50 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): dump: 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.374 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2350792345 0 0) 0x7fc9c8004f10 con 0x7fc9e00a0570 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9e0120490 con 0x7fc9e00a0570 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc9cc001d10 con 0x7fc9e00a0570 2024-09-18T11:49:54.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient(hunting): dump: 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2018897124 0 0) 0x7fc9e013dd90 con 0x7fc9e0097a50 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.112:6790/0 conn(0x7fc9e00a0570 legacy=0x7fc9e01468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6789/0 conn(0x7fc9e00a1050 legacy=0x7fc9e014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient: _finish_hunting 0 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 monclient: found mon.a 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e0122720 con 0x7fc9e0097a50 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient: _finish_auth 0 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 5 monclient: authenticate success, global_id 94119 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 10 monclient: _renew_subs 2024-09-18T11:49:54.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e0122ce0 con 0x7fc9e0097a50 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 10 client.94119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 10 monclient: _renew_subs 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e013eb80 con 0x7fc9e0097a50 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9e6416700 1 librados: init done 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d0004f00 con 0x7fc9e0097a50 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc9d0005210 con 0x7fc9e0097a50 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9ddffb700 4 set_mon_vals no callback set 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.357+0000 7fc9d7fff700 10 monclient: dump: 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.379 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9e6416700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc9bc006590 con 0x7fc9bc002840 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fc9d000fa90 con 0x7fc9e0097a50 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(259..259 src has 1..259) v4 ==== 5676+0+0 (unknown 340334306 0 0) 0x7fc9d0043fe0 con 0x7fc9e0097a50 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 client.94119.objecter ms_dispatch 0x7fc9e0000fd0 osd_map(259..259 src has 1..259) v4 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 3 client.94119.objecter handle_osd_map got epochs [259,259] > 0 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 3 client.94119.objecter handle_osd_map decoding full epoch 259 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 20 client.94119.objecter dump_active .. 0 homeless 2024-09-18T11:49:54.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9e49b3700 10 monclient: get_auth_request con 0x7fc9c8038290 auth_method 0 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 client.94119.objecter ms_handle_connect 0x7fc9bc002840 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 client.94119.objecter resend_mon_ops 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9df7fe700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6789/0 conn(0x7fc9bc002840 legacy=0x7fc9bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339196929 0 0) 0x7fc9bc006590 con 0x7fc9bc002840 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 monclient(hunting): my global_id is 18040 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e013dd90 con 0x7fc9bc002840 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 client.94119.objecter ms_handle_connect 0x7fc9c8038290 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 734323791 0 0) 0x7fc9e013dd90 con 0x7fc9bc002840 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9bc006590 con 0x7fc9bc002840 2024-09-18T11:49:54.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc9cc004cf0 con 0x7fc9bc002840 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.361+0000 7fc9d7fff700 10 monclient: dump: 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.382 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.365+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1127734068 0 0) 0x7fc9bc006590 con 0x7fc9bc002840 2024-09-18T11:49:54.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.365+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9bc00a140 con 0x7fc9bc002840 2024-09-18T11:49:54.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.365+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fc9cc005920 con 0x7fc9bc002840 2024-09-18T11:49:54.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.365+0000 7fc9d7fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.365+0000 7fc9d7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.365+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6789/0 conn(0x7fc9bc002840 legacy=0x7fc9bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.365+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6789/0 conn(0x7fc9bc002840 legacy=0x7fc9bc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9e6416700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc9bc009670 con 0x7fc9bc0090e0 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9df7fe700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6789/0 conn(0x7fc9bc0090e0 legacy=0x7fc9bc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 10 client.94119.objecter ms_handle_connect 0x7fc9bc0090e0 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 10 client.94119.objecter resend_mon_ops 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371031082 0 0) 0x7fc9bc009670 con 0x7fc9bc0090e0 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 10 monclient(hunting): my global_id is 18043 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9bc006590 con 0x7fc9bc0090e0 2024-09-18T11:49:54.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3887104324 0 0) 0x7fc9bc006590 con 0x7fc9bc0090e0 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9bc009670 con 0x7fc9bc0090e0 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc9cc0053a0 con 0x7fc9bc0090e0 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 10 monclient: dump: 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.387 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3498610416 0 0) 0x7fc9bc009670 con 0x7fc9bc0090e0 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.377+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc9bc009930 con 0x7fc9bc0090e0 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 4045479989) 0x7fc9cc005670 con 0x7fc9bc0090e0 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9d7fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9d7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6789/0 conn(0x7fc9bc0090e0 legacy=0x7fc9bc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9d7fff700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6789/0 conn(0x7fc9bc0090e0 legacy=0x7fc9bc008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 28, 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:49:54.389 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 1, 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:54.390 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:49:54.392 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:54.393 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:49:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:54.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.397 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:54.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:54.399 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:49:54.400 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:49:54.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 10 librados: watch_flush enter 2024-09-18T11:49:54.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 10 librados: watch_flush exit 2024-09-18T11:49:54.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 20 client.94119.objecter shutdown clearing up homeless session... 2024-09-18T11:49:54.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 10 client.94119.objecter successfully canceled tick 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.129:6812/20806 conn(0x7fc9c8038290 legacy=0x7fc9c803a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 10 monclient: shutdown 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 >> v1:172.21.15.112:6789/0 conn(0x7fc9e0097a50 legacy=0x7fc9e013d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e49b3700 1 -- 172.21.15.112:0/1383900774 reap_dead start 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e49b3700 1 -- 172.21.15.112:0/1383900774 reap_dead start 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 shutdown_connections 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 >> 172.21.15.112:0/1383900774 conn(0x7fc9e0067680 msgr2=0x7fc9e00985d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 shutdown_connections 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 1 -- 172.21.15.112:0/1383900774 wait complete. 2024-09-18T11:49:54.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.381+0000 7fc9e6416700 1 librados: shutdown 2024-09-18T11:49:54.404 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:49:54.404 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T11:49:54.404 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient: get_monmap_and_config 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient: build_initial_monmap 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient: monmap: 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:54.512820+0000 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:54.512820+0000 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:54.514 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 1 Processor -- start 2024-09-18T11:49:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 1 -- start start 2024-09-18T11:49:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient: init 2024-09-18T11:49:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:49:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient(hunting): picked mon.noname-b con 0x7ff0900a0500 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient(hunting): picked mon.noname-c con 0x7ff0900a0fe0 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient(hunting): picked mon.noname-a con 0x7ff090097ac0 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:54.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0900a13e0 con 0x7ff090097ac0 2024-09-18T11:49:54.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0900a3370 con 0x7ff0900a0fe0 2024-09-18T11:49:54.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0900a1660 con 0x7ff0900a0500 2024-09-18T11:49:54.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:54.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff0954a1700 10 monclient(hunting): authenticate will time out at 1166.354513s 2024-09-18T11:49:54.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff08effd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff0900a0500 0x7ff0900a08d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59850/0 (socket says 172.21.15.112:59850) 2024-09-18T11:49:54.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.509+0000 7ff08effd700 1 -- 172.21.15.112:0/1305018824 learned_addr learned my addr 172.21.15.112:0/1305018824 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:54.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08effd700 1 -- 172.21.15.112:0/1305018824 >> v1:172.21.15.112:6790/0 conn(0x7ff0900a0500 legacy=0x7ff0900a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08f7fe700 1 -- 172.21.15.112:0/1305018824 >> v1:172.21.15.112:6789/0 conn(0x7ff090097ac0 legacy=0x7ff09013ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08e7fc700 1 -- 172.21.15.112:0/1305018824 >> v1:172.21.15.129:6789/0 conn(0x7ff0900a0fe0 legacy=0x7ff090137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519920733 0 0) 0x7ff0900a1660 con 0x7ff0900a0500 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): my global_id is 84323 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff078003570 con 0x7ff0900a0500 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232881759 0 0) 0x7ff088001820 con 0x7ff090097ac0 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): my global_id is 94122 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0900a1660 con 0x7ff090097ac0 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846013797 0 0) 0x7ff0900a3370 con 0x7ff0900a0fe0 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient(hunting): my global_id is 18046 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff088001820 con 0x7ff0900a0fe0 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2920868648 0 0) 0x7ff078003570 con 0x7ff0900a0500 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 >> v1:172.21.15.129:6789/0 conn(0x7ff0900a0fe0 legacy=0x7ff090137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 >> v1:172.21.15.112:6789/0 conn(0x7ff090097ac0 legacy=0x7ff09013ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: _finish_hunting 0 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 monclient: found mon.noname-b 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff090096a60 con 0x7ff0900a0500 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: _finish_auth 0 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 5 monclient: authenticate success, global_id 84323 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff080001a00 con 0x7ff0900a0500 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: dump: 2024-09-18T11:49:54.522 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff080001b60 con 0x7ff0900a0500 2024-09-18T11:49:54.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 1 -- 172.21.15.112:0/1305018824 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff0800048f0 con 0x7ff0900a0500 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 4 set_mon_vals no callback set 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff08dffb700 10 monclient: dump: 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.524 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 10 monclient: shutdown 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 1 -- 172.21.15.112:0/1305018824 >> v1:172.21.15.112:6790/0 conn(0x7ff0900a0500 legacy=0x7ff0900a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 1 -- 172.21.15.112:0/1305018824 shutdown_connections 2024-09-18T11:49:54.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 1 -- 172.21.15.112:0/1305018824 >> 172.21.15.112:0/1305018824 conn(0x7ff09006a9b0 msgr2=0x7ff090067680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:54.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 1 -- 172.21.15.112:0/1305018824 shutdown_connections 2024-09-18T11:49:54.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.513+0000 7ff0954a1700 1 -- 172.21.15.112:0/1305018824 wait complete. 2024-09-18T11:49:54.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient: build_initial_monmap 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient: monmap: 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 librados: starting msgr at 2024-09-18T11:49:54.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 librados: starting objecter 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 Processor -- start 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 -- start start 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 librados: setting wanted keys 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 librados: calling monclient init 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient: init 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient(hunting): picked mon.noname-a con 0x7ff090097ac0 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient(hunting): picked mon.noname-c con 0x7ff0900a0500 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient(hunting): picked mon.noname-b con 0x7ff0900a0fe0 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff090122c90 con 0x7ff090097ac0 2024-09-18T11:49:54.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09013dd40 con 0x7ff0900a0fe0 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff090120ca0 con 0x7ff0900a0500 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff0954a1700 10 monclient(hunting): authenticate will time out at 1166.361657s 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff08e7fc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff0900a0500 0x7ff090146880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59862/0 (socket says 172.21.15.112:59862) 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff08e7fc700 1 -- 172.21.15.112:0/908493264 learned_addr learned my addr 172.21.15.112:0/908493264 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff08e7fc700 1 -- 172.21.15.112:0/908493264 >> v1:172.21.15.112:6790/0 conn(0x7ff0900a0500 legacy=0x7ff090146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff076ffd700 10 client.?.objecter ms_handle_connect 0x7ff0900a0500 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff076ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff076ffd700 10 client.?.objecter ms_handle_connect 0x7ff090097ac0 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff076ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.517+0000 7ff08effd700 1 -- 172.21.15.112:0/908493264 >> v1:172.21.15.112:6789/0 conn(0x7ff090097ac0 legacy=0x7ff09013d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001241806 0 0) 0x7ff090122c90 con 0x7ff090097ac0 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): my global_id is 94125 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff08f7fe700 1 -- 172.21.15.112:0/908493264 >> v1:172.21.15.129:6789/0 conn(0x7ff0900a0fe0 legacy=0x7ff090149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c004f10 con 0x7ff090097ac0 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 client.?.objecter ms_handle_connect 0x7ff0900a0fe0 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524450338 0 0) 0x7ff090120ca0 con 0x7ff0900a0500 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): my global_id is 84326 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff090122c90 con 0x7ff0900a0500 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603542008 0 0) 0x7ff09013dd40 con 0x7ff0900a0fe0 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): my global_id is 18049 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff090120ca0 con 0x7ff0900a0fe0 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 283627361 0 0) 0x7ff07c004f10 con 0x7ff090097ac0 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff09013dd40 con 0x7ff090097ac0 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff080004830 con 0x7ff090097ac0 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): dump: 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.536 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3123144913 0 0) 0x7ff090122c90 con 0x7ff0900a0500 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff07c004f10 con 0x7ff0900a0500 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff084001cb0 con 0x7ff0900a0500 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): dump: 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2195208357 0 0) 0x7ff090120ca0 con 0x7ff0900a0fe0 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff090122c90 con 0x7ff0900a0fe0 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff088002280 con 0x7ff0900a0fe0 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient(hunting): dump: 2024-09-18T11:49:54.538 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1128928189 0 0) 0x7ff09013dd40 con 0x7ff090097ac0 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 >> v1:172.21.15.112:6790/0 conn(0x7ff0900a0500 legacy=0x7ff090146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 >> v1:172.21.15.129:6789/0 conn(0x7ff0900a0fe0 legacy=0x7ff090149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 monclient: found mon.a 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0901204d0 con 0x7ff090097ac0 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient: _finish_auth 0 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 5 monclient: authenticate success, global_id 94125 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 10 monclient: _renew_subs 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0901226d0 con 0x7ff090097ac0 2024-09-18T11:49:54.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 10 client.94125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 10 monclient: _renew_subs 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09013eb30 con 0x7ff090097ac0 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 1 librados: init done 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff080004f50 con 0x7ff090097ac0 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff0800015c0 con 0x7ff090097ac0 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff08d7fa700 4 set_mon_vals no callback set 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 10 monclient: dump: 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:54.541 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0700050b0 con 0x7ff090097ac0 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.521+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ff08000f980 con 0x7ff090097ac0 2024-09-18T11:49:54.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.525+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(259..259 src has 1..259) v4 ==== 5676+0+0 (unknown 340334306 0 0) 0x7ff080044040 con 0x7ff090097ac0 2024-09-18T11:49:54.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.525+0000 7ff076ffd700 10 client.94125.objecter ms_dispatch 0x7ff090000fd0 osd_map(259..259 src has 1..259) v4 2024-09-18T11:49:54.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.525+0000 7ff076ffd700 3 client.94125.objecter handle_osd_map got epochs [259,259] > 0 2024-09-18T11:49:54.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.525+0000 7ff076ffd700 3 client.94125.objecter handle_osd_map decoding full epoch 259 2024-09-18T11:49:54.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.525+0000 7ff08e7fc700 10 monclient: get_auth_request con 0x7ff07c038570 auth_method 0 2024-09-18T11:49:54.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.525+0000 7ff076ffd700 20 client.94125.objecter dump_active .. 0 homeless 2024-09-18T11:49:54.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.525+0000 7ff076ffd700 10 client.94125.objecter ms_handle_connect 0x7ff07c038570 2024-09-18T11:49:54.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.533+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7ff080014c20 con 0x7ff090097ac0 2024-09-18T11:49:54.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.533+0000 7ff076ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:54.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.533+0000 7ff076ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:54.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.757+0000 7ff0954a1700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T11:49:54.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.757+0000 7ff0954a1700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T11:49:54.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.757+0000 7ff0954a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:54.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.757+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7ff070005c30 con 0x7ff090097ac0 2024-09-18T11:49:54.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.761+0000 7ff076ffd700 1 -- 172.21.15.112:0/908493264 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7ff080014620 con 0x7ff090097ac0 2024-09-18T11:49:54.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.761+0000 7ff076ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T11:49:54.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.761+0000 7ff076ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:54.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.761+0000 7ff0954a1700 10 librados: watch_flush enter 2024-09-18T11:49:54.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.761+0000 7ff0954a1700 10 librados: watch_flush exit 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.761+0000 7ff0954a1700 20 client.94125.objecter shutdown clearing up homeless session... 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.761+0000 7ff0954a1700 10 client.94125.objecter successfully canceled tick 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 >> v1:172.21.15.129:6812/20806 conn(0x7ff07c038570 legacy=0x7ff07c03aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 10 monclient: shutdown 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 >> v1:172.21.15.112:6789/0 conn(0x7ff090097ac0 legacy=0x7ff09013d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 shutdown_connections 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 >> 172.21.15.112:0/908493264 conn(0x7ff09006a9b0 msgr2=0x7ff0900a48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 shutdown_connections 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 1 -- 172.21.15.112:0/908493264 wait complete. 2024-09-18T11:49:54.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:54.765+0000 7ff0954a1700 1 librados: shutdown 2024-09-18T11:49:54.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T11:49:54.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:54.797+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:54.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:54.797+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=7,logm=64} crc {auth=2724718272,config=3140514009,health=1340664802,kv=3304318502,logm=1007382447}) 2024-09-18T11:49:54.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:54.797+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=7,logm=63} crc {auth=125083537,config=3140514009,health=4000123824,kv=2703126420,logm=901291025}) 2024-09-18T11:49:54.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:54.797+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:54.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:54.797+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=7,logm=64} crc {auth=2724718272,config=3140514009,health=1340664802,kv=3304318502,logm=1007382447}) 2024-09-18T11:49:54.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:54.797+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=8,logm=63} crc {auth=3971696405,config=3140514009,health=4000123824,kv=1310296871,logm=1366743743}) 2024-09-18T11:49:55.108 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.109 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353216921}) 2024-09-18T11:49:55.109 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177504615}) 2024-09-18T11:49:55.109 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.109 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353216921}) 2024-09-18T11:49:55.109 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342292668}) 2024-09-18T11:49:55.111 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.111 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428428862}) 2024-09-18T11:49:55.112 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=593597650}) 2024-09-18T11:49:55.112 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.112 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428428862}) 2024-09-18T11:49:55.112 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.105+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152433260}) 2024-09-18T11:49:55.114 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.109+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.114 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.109+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152306337}) 2024-09-18T11:49:55.114 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.109+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2982973011}) 2024-09-18T11:49:55.114 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.109+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.114 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.109+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152306337}) 2024-09-18T11:49:55.114 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.109+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282769881}) 2024-09-18T11:49:55.116 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.117 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3629697650}) 2024-09-18T11:49:55.117 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2990330537}) 2024-09-18T11:49:55.117 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.117 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3629697650}) 2024-09-18T11:49:55.117 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3101723654}) 2024-09-18T11:49:55.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772068598}) 2024-09-18T11:49:55.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970145485}) 2024-09-18T11:49:55.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772068598}) 2024-09-18T11:49:55.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.113+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=444432659}) 2024-09-18T11:49:55.121 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.121 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2685042600}) 2024-09-18T11:49:55.121 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919080347}) 2024-09-18T11:49:55.121 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.121 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2685042600}) 2024-09-18T11:49:55.122 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842363405}) 2024-09-18T11:49:55.123 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.123 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011429417}) 2024-09-18T11:49:55.123 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=888043948}) 2024-09-18T11:49:55.123 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.123 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011429417}) 2024-09-18T11:49:55.123 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.117+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3079697634}) 2024-09-18T11:49:55.124 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731144365}) 2024-09-18T11:49:55.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3600685331}) 2024-09-18T11:49:55.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731144365}) 2024-09-18T11:49:55.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624902256}) 2024-09-18T11:49:55.126 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.126 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577838422}) 2024-09-18T11:49:55.126 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2323384388}) 2024-09-18T11:49:55.126 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.126 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577838422}) 2024-09-18T11:49:55.126 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1756790413}) 2024-09-18T11:49:55.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610068643}) 2024-09-18T11:49:55.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4072674816}) 2024-09-18T11:49:55.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610068643}) 2024-09-18T11:49:55.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.121+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=698191764}) 2024-09-18T11:49:55.176 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.169+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.176 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.169+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028569766}) 2024-09-18T11:49:55.176 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.169+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2815684103}) 2024-09-18T11:49:55.176 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.169+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.176 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.169+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028569766}) 2024-09-18T11:49:55.176 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.169+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3913110602}) 2024-09-18T11:49:55.183 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.177+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.183 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.177+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895599073}) 2024-09-18T11:49:55.183 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.177+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2338845558}) 2024-09-18T11:49:55.183 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.177+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.184 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.177+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895599073}) 2024-09-18T11:49:55.184 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.177+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282852845}) 2024-09-18T11:49:55.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.185+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.185+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977474429}) 2024-09-18T11:49:55.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.185+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4285378842}) 2024-09-18T11:49:55.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.185+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.185+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977474429}) 2024-09-18T11:49:55.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.185+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942663894}) 2024-09-18T11:49:55.198 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.193+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.198 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.193+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224295363}) 2024-09-18T11:49:55.198 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.193+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251743471}) 2024-09-18T11:49:55.198 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.193+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.198 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.193+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224295363}) 2024-09-18T11:49:55.198 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.193+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534435959}) 2024-09-18T11:49:55.282 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.277+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.282 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.277+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151231390}) 2024-09-18T11:49:55.282 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.277+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3133507202}) 2024-09-18T11:49:55.282 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.277+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.282 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.277+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151231390}) 2024-09-18T11:49:55.282 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.277+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236567294}) 2024-09-18T11:49:55.284 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.281+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.284 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.281+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181164209}) 2024-09-18T11:49:55.284 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.281+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3595538255}) 2024-09-18T11:49:55.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.281+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:55.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.281+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181164209}) 2024-09-18T11:49:55.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.281+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3055383387}) 2024-09-18T11:49:55.784 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:49:55.784 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:49:55.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.889+0000 7f107f42a700 10 monclient: get_monmap_and_config 2024-09-18T11:49:55.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.889+0000 7f107f42a700 10 monclient: build_initial_monmap 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.889+0000 7f107f42a700 10 monclient: monmap: 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:55.894306+0000 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:55.894306+0000 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:55.895 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:55.896 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:55.896 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.889+0000 7f107f42a700 1 Processor -- start 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.889+0000 7f107f42a700 1 -- start start 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.889+0000 7f107f42a700 10 monclient: init 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 10 monclient(hunting): picked mon.noname-a con 0x7f1078067680 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 10 monclient(hunting): picked mon.noname-c con 0x7f10780994f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 10 monclient(hunting): picked mon.noname-b con 0x7f1078097880 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:55.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1078098640 con 0x7f1078067680 2024-09-18T11:49:55.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1078067a70 con 0x7f10780994f0 2024-09-18T11:49:55.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107809dee0 con 0x7f1078097880 2024-09-18T11:49:55.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:55.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 10 monclient(hunting): authenticate will time out at 1167.736701s 2024-09-18T11:49:55.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107e428700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1078067680 0x7f1078096a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54060/0 (socket says 172.21.15.112:54060) 2024-09-18T11:49:55.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107e428700 1 -- 172.21.15.112:0/2186587221 learned_addr learned my addr 172.21.15.112:0/2186587221 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:55.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107ec29700 1 -- 172.21.15.112:0/2186587221 >> v1:172.21.15.112:6790/0 conn(0x7f1078097880 legacy=0x7f1078137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:55.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107e428700 1 -- 172.21.15.112:0/2186587221 >> v1:172.21.15.112:6789/0 conn(0x7f1078067680 legacy=0x7f1078096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:55.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541199413 0 0) 0x7f1078098640 con 0x7f1078067680 2024-09-18T11:49:55.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:55.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:55.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient(hunting): my global_id is 94128 2024-09-18T11:49:55.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1064003570 con 0x7f1078067680 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407444605 0 0) 0x7f107809dee0 con 0x7f1078097880 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient(hunting): my global_id is 84329 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1078098640 con 0x7f1078097880 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107dc27700 1 -- 172.21.15.112:0/2186587221 >> v1:172.21.15.129:6789/0 conn(0x7f10780994f0 legacy=0x7f1078097170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716002450 0 0) 0x7f1078098640 con 0x7f1078097880 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 >> v1:172.21.15.129:6789/0 conn(0x7f10780994f0 legacy=0x7f1078097170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 >> v1:172.21.15.112:6789/0 conn(0x7f1078067680 legacy=0x7f1078096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient: _finish_hunting 0 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 monclient: found mon.noname-b 2024-09-18T11:49:55.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078098c00 con 0x7f1078097880 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient: _finish_auth 0 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 5 monclient: authenticate success, global_id 84329 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1070001db0 con 0x7f1078097880 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107d426700 10 monclient: dump: 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.903 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.893+0000 7f107f42a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1070001f90 con 0x7f1078097880 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107d426700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 4 set_mon_vals no callback set 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107d426700 1 -- 172.21.15.112:0/2186587221 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1070002420 con 0x7f1078097880 2024-09-18T11:49:55.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107d426700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107d426700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107d426700 10 monclient: dump: 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:55.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient: shutdown 2024-09-18T11:49:55.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:55.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- 172.21.15.112:0/2186587221 >> v1:172.21.15.112:6790/0 conn(0x7f1078097880 legacy=0x7f1078137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:55.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- 172.21.15.112:0/2186587221 shutdown_connections 2024-09-18T11:49:55.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- 172.21.15.112:0/2186587221 >> 172.21.15.112:0/2186587221 conn(0x7f10780a1e30 msgr2=0x7f10780a2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:55.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- 172.21.15.112:0/2186587221 shutdown_connections 2024-09-18T11:49:55.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- 172.21.15.112:0/2186587221 wait complete. 2024-09-18T11:49:55.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient: build_initial_monmap 2024-09-18T11:49:55.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient: monmap: 2024-09-18T11:49:55.913 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:55.913 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 librados: starting msgr at 2024-09-18T11:49:55.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 librados: starting objecter 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 Processor -- start 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- start start 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 librados: setting wanted keys 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 librados: calling monclient init 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient: init 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient(hunting): picked mon.noname-b con 0x7f1078067680 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient(hunting): picked mon.noname-a con 0x7f1078097880 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient(hunting): picked mon.noname-c con 0x7f10780994f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107809b040 con 0x7f1078097880 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107806c100 con 0x7f1078067680 2024-09-18T11:49:55.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107806d230 con 0x7f10780994f0 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107f42a700 10 monclient(hunting): authenticate will time out at 1167.743120s 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107ec29700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f10780994f0 0x7f1078149760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59882/0 (socket says 172.21.15.112:59882) 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.897+0000 7f107ec29700 1 -- 172.21.15.112:0/2930094036 learned_addr learned my addr 172.21.15.112:0/2930094036 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107dc27700 1 -- 172.21.15.112:0/2930094036 >> v1:172.21.15.112:6789/0 conn(0x7f1078097880 legacy=0x7f107806b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 client.?.objecter ms_handle_connect 0x7f1078097880 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 client.?.objecter ms_handle_connect 0x7f10780994f0 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107ec29700 1 -- 172.21.15.112:0/2930094036 >> v1:172.21.15.112:6790/0 conn(0x7f10780994f0 legacy=0x7f1078149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117462268 0 0) 0x7f107806d230 con 0x7f10780994f0 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:55.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): my global_id is 84332 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105c004f10 con 0x7f10780994f0 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216406655 0 0) 0x7f107809b040 con 0x7f1078097880 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): my global_id is 94131 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107806d230 con 0x7f1078097880 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 client.?.objecter ms_handle_connect 0x7f1078067680 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107e428700 1 -- 172.21.15.112:0/2930094036 >> v1:172.21.15.129:6789/0 conn(0x7f1078067680 legacy=0x7f107806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428291753 0 0) 0x7f107806c100 con 0x7f1078067680 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): my global_id is 18055 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107809b040 con 0x7f1078067680 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3220966004 0 0) 0x7f105c004f10 con 0x7f10780994f0 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f107806c100 con 0x7f10780994f0 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 89264425 0 0) 0x7f107806d230 con 0x7f1078097880 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f105c004f10 con 0x7f1078097880 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1070001cf0 con 0x7f10780994f0 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): dump: 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:55.918 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1068001b40 con 0x7f1078097880 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient(hunting): dump: 2024-09-18T11:49:55.919 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4291597138 0 0) 0x7f107806c100 con 0x7f10780994f0 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 >> v1:172.21.15.129:6789/0 conn(0x7f1078067680 legacy=0x7f107806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 >> v1:172.21.15.112:6789/0 conn(0x7f1078097880 legacy=0x7f107806b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:55.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 monclient: found mon.c 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078097ca0 con 0x7f10780994f0 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 5 monclient: authenticate success, global_id 84332 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 10 monclient: _renew_subs 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107806cd20 con 0x7f10780994f0 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 10 client.84332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 10 monclient: _renew_subs 2024-09-18T11:49:55.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f107806d8f0 con 0x7f10780994f0 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 1 librados: init done 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1070002a80 con 0x7f10780994f0 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1070002e10 con 0x7f10780994f0 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107cc25700 4 set_mon_vals no callback set 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f106e7fc700 10 monclient: dump: 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:55.922 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.901+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10600050b0 con 0x7f10780994f0 2024-09-18T11:49:55.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.905+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f10700102b0 con 0x7f10780994f0 2024-09-18T11:49:55.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.905+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(260..260 src has 1..260) v4 ==== 5439+0+0 (unknown 3298675348 0 0) 0x7f1070044340 con 0x7f10780994f0 2024-09-18T11:49:55.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.905+0000 7f106e7fc700 10 client.84332.objecter ms_dispatch 0x7f1078000fd0 osd_map(260..260 src has 1..260) v4 2024-09-18T11:49:55.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.905+0000 7f106e7fc700 3 client.84332.objecter handle_osd_map got epochs [260,260] > 0 2024-09-18T11:49:55.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.905+0000 7f106e7fc700 3 client.84332.objecter handle_osd_map decoding full epoch 260 2024-09-18T11:49:55.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.905+0000 7f106e7fc700 20 client.84332.objecter dump_active .. 0 homeless 2024-09-18T11:49:55.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.905+0000 7f107e428700 10 monclient: get_auth_request con 0x7f105c038580 auth_method 0 2024-09-18T11:49:55.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.905+0000 7f106e7fc700 10 client.84332.objecter ms_handle_connect 0x7f105c038580 2024-09-18T11:49:55.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.913+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f1070014fa0 con 0x7f10780994f0 2024-09-18T11:49:55.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.913+0000 7f106e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:55.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:55.913+0000 7f106e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:56.002 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.997+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.002 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.997+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874845658}) 2024-09-18T11:49:56.002 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.997+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803832153}) 2024-09-18T11:49:56.002 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.997+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.002 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.997+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874845658}) 2024-09-18T11:49:56.003 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:55.997+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3694189839}) 2024-09-18T11:49:56.004 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.004 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756572302}) 2024-09-18T11:49:56.004 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=453972151}) 2024-09-18T11:49:56.004 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.005 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756572302}) 2024-09-18T11:49:56.005 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466427745}) 2024-09-18T11:49:56.006 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.006 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945484836}) 2024-09-18T11:49:56.006 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1332459894}) 2024-09-18T11:49:56.006 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.006 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945484836}) 2024-09-18T11:49:56.006 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.001+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225915548}) 2024-09-18T11:49:56.008 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.008 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882072156}) 2024-09-18T11:49:56.008 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3291846951}) 2024-09-18T11:49:56.008 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.009 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882072156}) 2024-09-18T11:49:56.009 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=345128886}) 2024-09-18T11:49:56.010 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.010 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401304758}) 2024-09-18T11:49:56.010 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003996446}) 2024-09-18T11:49:56.010 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.011 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401304758}) 2024-09-18T11:49:56.011 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.005+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1987353231}) 2024-09-18T11:49:56.012 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.012 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293079068}) 2024-09-18T11:49:56.012 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616715382}) 2024-09-18T11:49:56.012 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.013 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293079068}) 2024-09-18T11:49:56.013 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2040210076}) 2024-09-18T11:49:56.014 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.014 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336345437}) 2024-09-18T11:49:56.014 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1062249285}) 2024-09-18T11:49:56.014 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.015 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336345437}) 2024-09-18T11:49:56.015 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.009+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392316360}) 2024-09-18T11:49:56.016 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.016 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485251430}) 2024-09-18T11:49:56.016 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266519660}) 2024-09-18T11:49:56.016 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.016 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485251430}) 2024-09-18T11:49:56.016 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911104071}) 2024-09-18T11:49:56.017 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.017 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180977560}) 2024-09-18T11:49:56.018 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045638767}) 2024-09-18T11:49:56.018 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.018 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180977560}) 2024-09-18T11:49:56.018 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046159662}) 2024-09-18T11:49:56.018 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.019 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572033599}) 2024-09-18T11:49:56.019 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=222647632}) 2024-09-18T11:49:56.019 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.019 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572033599}) 2024-09-18T11:49:56.019 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.013+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367498370}) 2024-09-18T11:49:56.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.025+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.025+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704772680}) 2024-09-18T11:49:56.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.025+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649061672}) 2024-09-18T11:49:56.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.025+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.025+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704772680}) 2024-09-18T11:49:56.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.025+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=430233233}) 2024-09-18T11:49:56.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=2617951119,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=2106702284}) 2024-09-18T11:49:56.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=1431886275,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1851401487}) 2024-09-18T11:49:56.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=2617951119,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=2106702284}) 2024-09-18T11:49:56.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=4048520661,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=195116254}) 2024-09-18T11:49:56.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=730254756,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=2635617177,osd_snap=2861234841,osdmap=207596304}) 2024-09-18T11:49:56.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=47} crc {mgrstat=1656402051,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=2635617177,osd_snap=563434531,osdmap=1928816362}) 2024-09-18T11:49:56.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=730254756,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=2635617177,osd_snap=2861234841,osdmap=207596304}) 2024-09-18T11:49:56.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.029+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=49} crc {mgrstat=233216479,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=2635617177,osd_snap=3694717623,osdmap=1331197226}) 2024-09-18T11:49:56.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.149+0000 7f107f42a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:49:56.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.149+0000 7f107f42a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:49:56.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.149+0000 7f107f42a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:56.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.149+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f10600058e0 con 0x7f10780994f0 2024-09-18T11:49:56.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.149+0000 7f106e7fc700 1 -- 172.21.15.112:0/2930094036 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3987094654) 0x7f10700149a0 con 0x7f10780994f0 2024-09-18T11:49:56.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.149+0000 7f106e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:49:56.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.149+0000 7f106e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:56.156 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:49:56.157 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.157 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3545321460}) 2024-09-18T11:49:56.157 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3113460295}) 2024-09-18T11:49:56.157 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.157 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3545321460}) 2024-09-18T11:49:56.157 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=245508637}) 2024-09-18T11:49:56.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 10 librados: watch_flush enter 2024-09-18T11:49:56.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 10 librados: watch_flush exit 2024-09-18T11:49:56.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 20 client.84332.objecter shutdown clearing up homeless session... 2024-09-18T11:49:56.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 10 client.84332.objecter successfully canceled tick 2024-09-18T11:49:56.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 >> v1:172.21.15.129:6812/20806 conn(0x7f105c038580 legacy=0x7f105c03aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 10 monclient: shutdown 2024-09-18T11:49:56.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:56.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 >> v1:172.21.15.112:6790/0 conn(0x7f10780994f0 legacy=0x7f1078149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 shutdown_connections 2024-09-18T11:49:56.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 >> 172.21.15.112:0/2930094036 conn(0x7f10780a1e30 msgr2=0x7f1078134dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:56.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 shutdown_connections 2024-09-18T11:49:56.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 1 -- 172.21.15.112:0/2930094036 wait complete. 2024-09-18T11:49:56.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.153+0000 7f107f42a700 1 librados: shutdown 2024-09-18T11:49:56.171 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:49:56.171 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T11:49:56.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:49:56.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:49:56.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.189+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.189+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3537863102}) 2024-09-18T11:49:56.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.189+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1371673557}) 2024-09-18T11:49:56.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.189+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.189+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3537863102}) 2024-09-18T11:49:56.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.189+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=476250906}) 2024-09-18T11:49:56.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.197+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.197+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2724080616}) 2024-09-18T11:49:56.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.197+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=255073040}) 2024-09-18T11:49:56.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.197+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.197+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2724080616}) 2024-09-18T11:49:56.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.197+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=775634981}) 2024-09-18T11:49:56.223 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.217+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.217+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=38} crc {osdmap=2220088129}) 2024-09-18T11:49:56.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.217+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=39} crc {osdmap=2787244928}) 2024-09-18T11:49:56.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.217+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:49:56.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.217+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=38} crc {osdmap=2220088129}) 2024-09-18T11:49:56.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:49:56.217+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1537490142}) 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:7462ddf6:::16o.RoundTripAppendPP (3019 ms) 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3017 ms) 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3009 ms) 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2985 ms) 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: setting pool EIO 2024-09-18T11:49:56.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: max_success 1131, min_failed 1132 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4026 ms) 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3019 ms) 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (131227 ms total) 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3003 ms) 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-18T11:49:56.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8946 ms) 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14997 ms) 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3398 ms) 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30344 ms total) 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (17364 ms) 2024-09-18T11:49:56.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (17364 ms total) 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13516 ms) 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7386 ms) 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6599 ms) 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7025 ms) 2024-09-18T11:49:56.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2981 ms) 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7053 ms) 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6984 ms) 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6969 ms) 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7000 ms) 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7767 ms) 2024-09-18T11:49:56.231 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-18T11:49:56.232 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6248 ms) 2024-09-18T11:49:56.232 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-18T11:49:56.232 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6993 ms) 2024-09-18T11:49:56.232 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-18T11:49:56.232 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6914 ms) 2024-09-18T11:49:56.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient: get_monmap_and_config 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient: build_initial_monmap 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient: monmap: 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:56.288820+0000 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:56.288820+0000 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:56.289 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 1 Processor -- start 2024-09-18T11:49:56.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 1 -- start start 2024-09-18T11:49:56.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient: init 2024-09-18T11:49:56.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:56.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:49:56.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient(hunting): picked mon.noname-b con 0x7f89a409c030 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:56.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient(hunting): picked mon.noname-a con 0x7f89a4131800 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:56.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient(hunting): picked mon.noname-c con 0x7f89a4135300 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:56.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a4098ab0 con 0x7f89a4131800 2024-09-18T11:49:56.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a4063be0 con 0x7f89a4135300 2024-09-18T11:49:56.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a409e2d0 con 0x7f89a409c030 2024-09-18T11:49:56.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:56.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89aa2c4700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f89a409c030 0x7f89a4099500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59892/0 (socket says 172.21.15.112:59892) 2024-09-18T11:49:56.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89ac528700 10 monclient(hunting): authenticate will time out at 1168.131089s 2024-09-18T11:49:56.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.285+0000 7f89aa2c4700 1 -- 172.21.15.112:0/4189739595 learned_addr learned my addr 172.21.15.112:0/4189739595 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:56.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89aa2c4700 1 -- 172.21.15.112:0/4189739595 >> v1:172.21.15.112:6790/0 conn(0x7f89a409c030 legacy=0x7f89a4099500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a9ac3700 1 -- 172.21.15.112:0/4189739595 >> v1:172.21.15.112:6789/0 conn(0x7f89a4131800 legacy=0x7f89a4133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868516230 0 0) 0x7f89a409e2d0 con 0x7f89a409c030 2024-09-18T11:49:56.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient(hunting): my global_id is 84341 2024-09-18T11:49:56.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8990003570 con 0x7f89a409c030 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505675755 0 0) 0x7f89a4098ab0 con 0x7f89a4131800 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient(hunting): my global_id is 94140 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a409e2d0 con 0x7f89a4131800 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89aaac5700 1 -- 172.21.15.112:0/4189739595 >> v1:172.21.15.129:6789/0 conn(0x7f89a4135300 legacy=0x7f89a41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381886412 0 0) 0x7f89a409e2d0 con 0x7f89a4131800 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 >> v1:172.21.15.112:6790/0 conn(0x7f89a409c030 legacy=0x7f89a4099500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 >> v1:172.21.15.129:6789/0 conn(0x7f89a4135300 legacy=0x7f89a41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: _finish_hunting 0 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 monclient: found mon.noname-a 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:56.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a409e040 con 0x7f89a4131800 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: _finish_auth 0 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8994001970 con 0x7f89a4131800 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 5 monclient: authenticate success, global_id 94140 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: dump: 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.297 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8994001c00 con 0x7f89a4131800 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 4 set_mon_vals no callback set 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 1 -- 172.21.15.112:0/4189739595 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f89940020a0 con 0x7f89a4131800 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89a92c2700 10 monclient: dump: 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 10 monclient: shutdown 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 -- 172.21.15.112:0/4189739595 >> v1:172.21.15.112:6789/0 conn(0x7f89a4131800 legacy=0x7f89a4133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 -- 172.21.15.112:0/4189739595 shutdown_connections 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 -- 172.21.15.112:0/4189739595 >> 172.21.15.112:0/4189739595 conn(0x7f89a4068b30 msgr2=0x7f89a40a2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 -- 172.21.15.112:0/4189739595 shutdown_connections 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 -- 172.21.15.112:0/4189739595 wait complete. 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 10 monclient: build_initial_monmap 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 10 monclient: monmap: 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.300 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:56.301 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:56.301 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:56.301 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 librados: starting msgr at 2024-09-18T11:49:56.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 librados: starting objecter 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 Processor -- start 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 -- start start 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 librados: setting wanted keys 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 1 librados: calling monclient init 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.289+0000 7f89ac528700 10 monclient: init 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient(hunting): picked mon.noname-a con 0x7f89a409c030 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient(hunting): picked mon.noname-b con 0x7f89a4131800 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient(hunting): picked mon.noname-c con 0x7f89a4135300 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a4097a70 con 0x7f89a409c030 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a40a04f0 con 0x7f89a4131800 2024-09-18T11:49:56.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a411a460 con 0x7f89a4135300 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient(hunting): authenticate will time out at 1168.136398s 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89aa2c4700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f89a409c030 0x7f89a406dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54086/0 (socket says 172.21.15.112:54086) 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89aa2c4700 1 -- 172.21.15.112:0/2848523372 learned_addr learned my addr 172.21.15.112:0/2848523372 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89aaac5700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6790/0 conn(0x7f89a4135300 legacy=0x7f89a41457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89aa2c4700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6789/0 conn(0x7f89a409c030 legacy=0x7f89a406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 client.?.objecter ms_handle_connect 0x7f89a4135300 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 client.?.objecter ms_handle_connect 0x7f89a409c030 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079643831 0 0) 0x7f89a4097a70 con 0x7f89a409c030 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): my global_id is 94143 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8988004f10 con 0x7f89a409c030 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677832364 0 0) 0x7f89a411a460 con 0x7f89a4135300 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): my global_id is 84344 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a4097a70 con 0x7f89a4135300 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 client.?.objecter ms_handle_connect 0x7f89a4131800 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89a9ac3700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.129:6789/0 conn(0x7f89a4131800 legacy=0x7f89a406e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 537222089 0 0) 0x7f8988004f10 con 0x7f89a409c030 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89a411a460 con 0x7f89a409c030 2024-09-18T11:49:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f89a0001cd0 con 0x7f89a409c030 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): dump: 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3567014464 0 0) 0x7f89a4097a70 con 0x7f89a4135300 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8988004f10 con 0x7f89a4135300 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f899c001f00 con 0x7f89a4135300 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient(hunting): dump: 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 447921724 0 0) 0x7f89a411a460 con 0x7f89a409c030 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6790/0 conn(0x7f89a4135300 legacy=0x7f89a41457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.129:6789/0 conn(0x7f89a4131800 legacy=0x7f89a406e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 monclient: found mon.a 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a411a720 con 0x7f89a409c030 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient: _finish_auth 0 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 5 monclient: authenticate success, global_id 94143 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient: _renew_subs 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a41189e0 con 0x7f89a409c030 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 client.94143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient: _renew_subs 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a41196f0 con 0x7f89a409c030 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89ac528700 1 librados: init done 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89a0004be0 con 0x7f89a409c030 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f89a0004f30 con 0x7f89a409c030 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f89a8ac1700 4 set_mon_vals no callback set 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.293+0000 7f899a7fc700 10 monclient: dump: 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.314 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f89a0005090 con 0x7f89a409c030 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f89ac528700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f898c006590 con 0x7f898c002840 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(261..261 src has 1..261) v4 ==== 5324+0+0 (unknown 3957991064 0 0) 0x7f89a0043f80 con 0x7f89a409c030 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 client.94143.objecter ms_dispatch 0x7f89a4000fd0 osd_map(261..261 src has 1..261) v4 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 3 client.94143.objecter handle_osd_map got epochs [261,261] > 0 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 3 client.94143.objecter handle_osd_map decoding full epoch 261 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 20 client.94143.objecter dump_active .. 0 homeless 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f89aaac5700 10 monclient: get_auth_request con 0x7f8988038240 auth_method 0 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 client.94143.objecter ms_handle_connect 0x7f898c002840 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 client.94143.objecter resend_mon_ops 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f89a9ac3700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6789/0 conn(0x7f898c002840 legacy=0x7f898c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292252665 0 0) 0x7f898c006590 con 0x7f898c002840 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 monclient(hunting): my global_id is 94146 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a411a460 con 0x7f898c002840 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 53805079 0 0) 0x7f89a411a460 con 0x7f898c002840 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f898c006590 con 0x7f898c002840 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8994002170 con 0x7f898c002840 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.297+0000 7f899a7fc700 10 monclient: dump: 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.316 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.301+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 863161206 0 0) 0x7f898c006590 con 0x7f898c002840 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.301+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f898c00a140 con 0x7f898c002840 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.301+0000 7f899a7fc700 10 client.94143.objecter ms_handle_connect 0x7f8988038240 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.301+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f8994001640 con 0x7f898c002840 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.301+0000 7f899a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:56.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.301+0000 7f899a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:56.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.301+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6789/0 conn(0x7f898c002840 legacy=0x7f898c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.301+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6789/0 conn(0x7f898c002840 legacy=0x7f898c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.309+0000 7f89ac528700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.309+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f898c009670 con 0x7f898c0090e0 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89a9ac3700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6789/0 conn(0x7f898c0090e0 legacy=0x7f898c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 client.94143.objecter ms_handle_connect 0x7f898c0090e0 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 client.94143.objecter resend_mon_ops 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183714929 0 0) 0x7f898c009670 con 0x7f898c0090e0 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 monclient(hunting): my global_id is 94149 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f898c006590 con 0x7f898c0090e0 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 666342449 0 0) 0x7f898c006590 con 0x7f898c0090e0 2024-09-18T11:49:56.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f898c009670 con 0x7f898c0090e0 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8994001970 con 0x7f898c0090e0 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 monclient: dump: 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.320 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 721721525 0 0) 0x7f898c009670 con 0x7f898c0090e0 2024-09-18T11:49:56.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f898c009930 con 0x7f898c0090e0 2024-09-18T11:49:56.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 3920537159) 0x7f8994002920 con 0x7f898c0090e0 2024-09-18T11:49:56.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:49:56.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:56.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6789/0 conn(0x7f898c0090e0 legacy=0x7f898c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f899a7fc700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6789/0 conn(0x7f898c0090e0 legacy=0x7f898c008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 28, 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:49:56.322 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.323 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:49:56.325 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.326 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.328 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:49:56.328 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:49:56.328 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.328 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.328 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:49:56.329 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:49:56.330 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.331 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.332 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:49:56.333 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:49:56.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 10 librados: watch_flush enter 2024-09-18T11:49:56.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 10 librados: watch_flush exit 2024-09-18T11:49:56.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 20 client.94143.objecter shutdown clearing up homeless session... 2024-09-18T11:49:56.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 10 client.94143.objecter successfully canceled tick 2024-09-18T11:49:56.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.129:6812/20806 conn(0x7f8988038240 legacy=0x7f898803a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 10 monclient: shutdown 2024-09-18T11:49:56.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:56.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 >> v1:172.21.15.112:6789/0 conn(0x7f89a409c030 legacy=0x7f89a406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89aaac5700 1 -- 172.21.15.112:0/2848523372 reap_dead start 2024-09-18T11:49:56.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 shutdown_connections 2024-09-18T11:49:56.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 >> 172.21.15.112:0/2848523372 conn(0x7f89a4068b30 msgr2=0x7f89a409af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:56.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 shutdown_connections 2024-09-18T11:49:56.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.313+0000 7f89ac528700 1 -- 172.21.15.112:0/2848523372 wait complete. 2024-09-18T11:49:56.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.317+0000 7f89ac528700 1 librados: shutdown 2024-09-18T11:49:56.336 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:49:56.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:49:56.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:49:56.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient: get_monmap_and_config 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient: build_initial_monmap 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient: monmap: 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:56.484876+0000 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:56.484876+0000 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:56.485 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 1 Processor -- start 2024-09-18T11:49:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 1 -- start start 2024-09-18T11:49:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient: init 2024-09-18T11:49:56.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:56.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:49:56.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient(hunting): picked mon.noname-c con 0x7f933006add0 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:56.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient(hunting): picked mon.noname-a con 0x7f9330132c30 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:56.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient(hunting): picked mon.noname-b con 0x7f9330133070 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:56.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93300a17f0 con 0x7f9330132c30 2024-09-18T11:49:56.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9330097010 con 0x7f933006add0 2024-09-18T11:49:56.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93300972f0 con 0x7f9330133070 2024-09-18T11:49:56.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:56.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f9334dd0700 10 monclient(hunting): authenticate will time out at 1168.326681s 2024-09-18T11:49:56.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f932dd9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9330132c30 0x7f9330137850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54118/0 (socket says 172.21.15.112:54118) 2024-09-18T11:49:56.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.481+0000 7f932dd9b700 1 -- 172.21.15.112:0/1085844013 learned_addr learned my addr 172.21.15.112:0/1085844013 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:56.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932dd9b700 1 -- 172.21.15.112:0/1085844013 >> v1:172.21.15.112:6789/0 conn(0x7f9330132c30 legacy=0x7f9330137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932e59c700 1 -- 172.21.15.112:0/1085844013 >> v1:172.21.15.129:6789/0 conn(0x7f933006add0 legacy=0x7f9330132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932ed9d700 1 -- 172.21.15.112:0/1085844013 >> v1:172.21.15.112:6790/0 conn(0x7f9330133070 legacy=0x7f933013af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426405633 0 0) 0x7f93300a17f0 con 0x7f9330132c30 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): my global_id is 94152 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9318003570 con 0x7f9330132c30 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422221859 0 0) 0x7f9330097010 con 0x7f933006add0 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): my global_id is 18070 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93300a17f0 con 0x7f933006add0 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171270209 0 0) 0x7f93300972f0 con 0x7f9330133070 2024-09-18T11:49:56.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient(hunting): my global_id is 84347 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9330097010 con 0x7f9330133070 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 900954955 0 0) 0x7f9330097010 con 0x7f9330133070 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 >> v1:172.21.15.129:6789/0 conn(0x7f933006add0 legacy=0x7f9330132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 >> v1:172.21.15.112:6789/0 conn(0x7f9330132c30 legacy=0x7f9330137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: _finish_hunting 0 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 monclient: found mon.noname-b 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93300975d0 con 0x7f9330133070 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: _finish_auth 0 2024-09-18T11:49:56.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f93200018a0 con 0x7f9330133070 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 5 monclient: authenticate success, global_id 84347 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: dump: 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.493 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9320001a00 con 0x7f9330133070 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 1 -- 172.21.15.112:0/1085844013 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9320001f00 con 0x7f9330133070 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f932d59a700 10 monclient: dump: 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.494 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.495 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.495 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.495 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.495 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.495 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:56.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 4 set_mon_vals no callback set 2024-09-18T11:49:56.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 10 monclient: shutdown 2024-09-18T11:49:56.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:56.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 1 -- 172.21.15.112:0/1085844013 >> v1:172.21.15.112:6790/0 conn(0x7f9330133070 legacy=0x7f933013af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 1 -- 172.21.15.112:0/1085844013 shutdown_connections 2024-09-18T11:49:56.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 1 -- 172.21.15.112:0/1085844013 >> 172.21.15.112:0/1085844013 conn(0x7f93300a5f90 msgr2=0x7f93300a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:56.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 1 -- 172.21.15.112:0/1085844013 shutdown_connections 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.485+0000 7f9334dd0700 1 -- 172.21.15.112:0/1085844013 wait complete. 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient: build_initial_monmap 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient: monmap: 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:56.501 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 librados: starting msgr at 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 librados: starting objecter 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 Processor -- start 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 -- start start 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 librados: setting wanted keys 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 librados: calling monclient init 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient: init 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient(hunting): picked mon.noname-c con 0x7f933006add0 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient(hunting): picked mon.noname-a con 0x7f9330132c30 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient(hunting): picked mon.noname-b con 0x7f9330133070 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:56.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93301325e0 con 0x7f9330132c30 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933011d400 con 0x7f9330133070 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9330142650 con 0x7f933006add0 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f9334dd0700 10 monclient(hunting): authenticate will time out at 1168.334695s 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f932e59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f933006add0 0x7f9330131640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59916/0 (socket says 172.21.15.112:59916) 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f932dd9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9330132c30 0x7f9330146b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54126/0 (socket says 172.21.15.112:54126) 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.489+0000 7f932e59c700 1 -- 172.21.15.112:0/1161375328 learned_addr learned my addr 172.21.15.112:0/1161375328 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f932dd9b700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6789/0 conn(0x7f9330132c30 legacy=0x7f9330146b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 client.?.objecter ms_handle_connect 0x7f9330132c30 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 client.?.objecter ms_handle_connect 0x7f933006add0 2024-09-18T11:49:56.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f932e59c700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6790/0 conn(0x7f933006add0 legacy=0x7f9330131640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f932ed9d700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.129:6789/0 conn(0x7f9330133070 legacy=0x7f933014a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 client.?.objecter ms_handle_connect 0x7f9330133070 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684267860 0 0) 0x7f9330142650 con 0x7f933006add0 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): my global_id is 84350 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9310004f10 con 0x7f933006add0 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292423577 0 0) 0x7f93301325e0 con 0x7f9330132c30 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): my global_id is 94155 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9330142650 con 0x7f9330132c30 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649084362 0 0) 0x7f933011d400 con 0x7f9330133070 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): my global_id is 18073 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93301325e0 con 0x7f9330133070 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1876936534 0 0) 0x7f9310004f10 con 0x7f933006add0 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f933011d400 con 0x7f933006add0 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754324945 0 0) 0x7f9330142650 con 0x7f9330132c30 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9310004f10 con 0x7f9330132c30 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9328002280 con 0x7f933006add0 2024-09-18T11:49:56.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): dump: 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9324001360 con 0x7f9330132c30 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient(hunting): dump: 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.507 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1008218173 0 0) 0x7f93301325e0 con 0x7f9330133070 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9330142650 con 0x7f9330133070 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3409178689 0 0) 0x7f933011d400 con 0x7f933006add0 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.129:6789/0 conn(0x7f9330133070 legacy=0x7f933014a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6789/0 conn(0x7f9330132c30 legacy=0x7f9330146b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 monclient: found mon.c 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933011e760 con 0x7f933006add0 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 5 monclient: authenticate success, global_id 84350 2024-09-18T11:49:56.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 10 monclient: _renew_subs 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933009e310 con 0x7f933006add0 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 10 client.84350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 10 monclient: _renew_subs 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9330131d50 con 0x7f933006add0 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 1 librados: init done 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93280027b0 con 0x7f933006add0 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9328002b00 con 0x7f933006add0 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931ffff700 4 set_mon_vals no callback set 2024-09-18T11:49:56.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f931e7fc700 10 monclient: dump: 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.510 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:56.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.493+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9314006590 con 0x7f9314002840 2024-09-18T11:49:56.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f9328002c60 con 0x7f933006add0 2024-09-18T11:49:56.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(261..261 src has 1..261) v4 ==== 5324+0+0 (unknown 3957991064 0 0) 0x7f9328044280 con 0x7f933006add0 2024-09-18T11:49:56.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 client.84350.objecter ms_dispatch 0x7f9330000fd0 osd_map(261..261 src has 1..261) v4 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 3 client.84350.objecter handle_osd_map got epochs [261,261] > 0 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 3 client.84350.objecter handle_osd_map decoding full epoch 261 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 20 client.84350.objecter dump_active .. 0 homeless 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f932ed9d700 10 monclient: get_auth_request con 0x7f9310038680 auth_method 0 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f932dd9b700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6790/0 conn(0x7f9314002840 legacy=0x7f9314004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 client.84350.objecter ms_handle_connect 0x7f9314002840 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 client.84350.objecter resend_mon_ops 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7605 0 0) 0x7f9314006590 con 0x7f9314002840 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 monclient(hunting): my global_id is 84353 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933011d400 con 0x7f9314002840 2024-09-18T11:49:56.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3901985634 0 0) 0x7f933011d400 con 0x7f9314002840 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9314006590 con 0x7f9314002840 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9324004ef0 con 0x7f9314002840 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 monclient: dump: 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.514 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 125322003 0 0) 0x7f9314006590 con 0x7f9314002840 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f931400a140 con 0x7f9314002840 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 client.84350.objecter ms_handle_connect 0x7f9310038680 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f93240051c0 con 0x7f9314002840 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6790/0 conn(0x7f9314002840 legacy=0x7f9314004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.497+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6790/0 conn(0x7f9314002840 legacy=0x7f9314004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:56.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.509+0000 7f9334dd0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:49:56.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.509+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9314009670 con 0x7f93140090e0 2024-09-18T11:49:56.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f932dd9b700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6790/0 conn(0x7f93140090e0 legacy=0x7f9314008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 client.84350.objecter ms_handle_connect 0x7f93140090e0 2024-09-18T11:49:56.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 client.84350.objecter resend_mon_ops 2024-09-18T11:49:56.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079374691 0 0) 0x7f9314009670 con 0x7f93140090e0 2024-09-18T11:49:56.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 monclient(hunting): my global_id is 84356 2024-09-18T11:49:56.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9314006590 con 0x7f93140090e0 2024-09-18T11:49:56.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3564691664 0 0) 0x7f9314006590 con 0x7f93140090e0 2024-09-18T11:49:56.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9314009670 con 0x7f93140090e0 2024-09-18T11:49:56.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9324005630 con 0x7f93140090e0 2024-09-18T11:49:56.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:49:56.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 monclient: dump: 2024-09-18T11:49:56.520 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.520 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.520 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.520 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.520 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.520 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.521 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.521 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.521 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.521 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2940310372 0 0) 0x7f9314009670 con 0x7f93140090e0 2024-09-18T11:49:56.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9314009930 con 0x7f93140090e0 2024-09-18T11:49:56.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 2320507077) 0x7f93240079e0 con 0x7f93140090e0 2024-09-18T11:49:56.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6790/0 conn(0x7f93140090e0 legacy=0x7f9314008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f931e7fc700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6790/0 conn(0x7f93140090e0 legacy=0x7f9314008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 28, 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:49:56.522 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.523 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.524 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:49:56.525 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.526 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.527 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:49:56.529 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:49:56.530 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.531 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:49:56.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:49:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:49:56.533 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:49:56.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 10 librados: watch_flush enter 2024-09-18T11:49:56.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 10 librados: watch_flush exit 2024-09-18T11:49:56.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 20 client.84350.objecter shutdown clearing up homeless session... 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 10 client.84350.objecter successfully canceled tick 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.129:6812/20806 conn(0x7f9310038680 legacy=0x7f931003ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 10 monclient: shutdown 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 >> v1:172.21.15.112:6790/0 conn(0x7f933006add0 legacy=0x7f9330131640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f932ed9d700 1 -- 172.21.15.112:0/1161375328 reap_dead start 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f932ed9d700 1 -- 172.21.15.112:0/1161375328 reap_dead start 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 shutdown_connections 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 >> 172.21.15.112:0/1161375328 conn(0x7f93300a5f90 msgr2=0x7f93300a04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 shutdown_connections 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 1 -- 172.21.15.112:0/1161375328 wait complete. 2024-09-18T11:49:56.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.513+0000 7f9334dd0700 1 librados: shutdown 2024-09-18T11:49:56.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:49:56.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:49:56.537 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:49:56.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.645+0000 7f0b78b5e700 10 monclient: get_monmap_and_config 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.645+0000 7f0b78b5e700 10 monclient: build_initial_monmap 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.645+0000 7f0b78b5e700 10 monclient: monmap: 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:49:56.649781+0000 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:49:56.649781+0000 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:49:56.650 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.645+0000 7f0b78b5e700 1 Processor -- start 2024-09-18T11:49:56.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.645+0000 7f0b78b5e700 1 -- start start 2024-09-18T11:49:56.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.645+0000 7f0b78b5e700 10 monclient: init 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 10 monclient(hunting): picked mon.noname-b con 0x7f0b740a1190 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 10 monclient(hunting): picked mon.noname-c con 0x7f0b7406eaa0 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 10 monclient(hunting): picked mon.noname-a con 0x7f0b7409dce0 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7409ff60 con 0x7f0b7409dce0 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7406dab0 con 0x7f0b7406eaa0 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7409f300 con 0x7f0b740a1190 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 10 monclient(hunting): authenticate will time out at 1168.492329s 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b72d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0b7409dce0 0x7f0b74138510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54138/0 (socket says 172.21.15.112:54138) 2024-09-18T11:49:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b72d9d700 1 -- 172.21.15.112:0/2672703352 learned_addr learned my addr 172.21.15.112:0/2672703352 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:56.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b7259c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0b740a1190 0x7f0b7409e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59940/0 (socket says 172.21.15.112:59940) 2024-09-18T11:49:56.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b7259c700 1 -- 172.21.15.112:0/2672703352 >> v1:172.21.15.112:6790/0 conn(0x7f0b740a1190 legacy=0x7f0b7409e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b72d9d700 1 -- 172.21.15.112:0/2672703352 >> v1:172.21.15.112:6789/0 conn(0x7f0b7409dce0 legacy=0x7f0b74138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b69d9b700 1 -- 172.21.15.112:0/2672703352 >> v1:172.21.15.129:6789/0 conn(0x7f0b7406eaa0 legacy=0x7f0b7409ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810697331 0 0) 0x7f0b7409f300 con 0x7f0b740a1190 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): my global_id is 84359 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b60003570 con 0x7f0b740a1190 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606243522 0 0) 0x7f0b7409ff60 con 0x7f0b7409dce0 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): my global_id is 94158 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7409f300 con 0x7f0b7409dce0 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071923752 0 0) 0x7f0b7406dab0 con 0x7f0b7406eaa0 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient(hunting): my global_id is 18076 2024-09-18T11:49:56.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7409ff60 con 0x7f0b7406eaa0 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474970027 0 0) 0x7f0b7409f300 con 0x7f0b7409dce0 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 >> v1:172.21.15.112:6790/0 conn(0x7f0b740a1190 legacy=0x7f0b7409e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 >> v1:172.21.15.129:6789/0 conn(0x7f0b7406eaa0 legacy=0x7f0b7409ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 monclient: found mon.noname-a 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b740a0160 con 0x7f0b7409dce0 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: _finish_auth 0 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0b6c001d60 con 0x7f0b7409dce0 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 5 monclient: authenticate success, global_id 94158 2024-09-18T11:49:56.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: dump: 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b6c001f40 con 0x7f0b7409dce0 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 10 monclient: get_monmap_and_config success 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 1 -- 172.21.15.112:0/2672703352 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0b6c0020f0 con 0x7f0b7409dce0 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b78b5e700 4 set_mon_vals no callback set 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.649+0000 7f0b71d9b700 10 monclient: dump: 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.660 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.661 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.661 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.661 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.661 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.661 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient: shutdown 2024-09-18T11:49:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:56.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- 172.21.15.112:0/2672703352 >> v1:172.21.15.112:6789/0 conn(0x7f0b7409dce0 legacy=0x7f0b74138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- 172.21.15.112:0/2672703352 shutdown_connections 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- 172.21.15.112:0/2672703352 >> 172.21.15.112:0/2672703352 conn(0x7f0b740a4570 msgr2=0x7f0b740a05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- 172.21.15.112:0/2672703352 shutdown_connections 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- 172.21.15.112:0/2672703352 wait complete. 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient: build_initial_monmap 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient: monmap: 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.665 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 librados: starting msgr at 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 librados: starting objecter 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 Processor -- start 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- start start 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 librados: setting wanted keys 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 librados: calling monclient init 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient: init 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient: _reopen_session rank -1 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient(hunting): picked mon.noname-c con 0x7f0b7406eaa0 addr v1:172.21.15.112:6790/0 2024-09-18T11:49:56.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient(hunting): picked mon.noname-a con 0x7f0b7409dce0 addr v1:172.21.15.112:6789/0 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient(hunting): picked mon.noname-b con 0x7f0b740a1190 addr v1:172.21.15.129:6789/0 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7406f0a0 con 0x7f0b7409dce0 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7409d340 con 0x7f0b740a1190 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b74123150 con 0x7f0b7406eaa0 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient(hunting): _renew_subs 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b78b5e700 10 monclient(hunting): authenticate will time out at 1168.498364s 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b69d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0b7409dce0 0x7f0b7411f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54150/0 (socket says 172.21.15.112:54150) 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b69d9b700 1 -- 172.21.15.112:0/708712874 learned_addr learned my addr 172.21.15.112:0/708712874 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.653+0000 7f0b7259c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0b7406eaa0 0x7f0b7411eed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59956/0 (socket says 172.21.15.112:59956) 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b69d9b700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.112:6789/0 conn(0x7f0b7409dce0 legacy=0x7f0b7411f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 client.?.objecter ms_handle_connect 0x7f0b7409dce0 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b7259c700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.112:6790/0 conn(0x7f0b7406eaa0 legacy=0x7f0b7411eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 client.?.objecter ms_handle_connect 0x7f0b7406eaa0 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131707708 0 0) 0x7f0b7406f0a0 con 0x7f0b7409dce0 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): my global_id is 94161 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b54004f10 con 0x7f0b7409dce0 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 DispatchQueue::entry inject delay of 0.002000 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b72d9d700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6789/0 conn(0x7f0b740a1190 legacy=0x7f0b7414a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 client.?.objecter ms_handle_connect 0x7f0b740a1190 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749930711 0 0) 0x7f0b74123150 con 0x7f0b7406eaa0 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): my global_id is 84362 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7406f0a0 con 0x7f0b7406eaa0 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1838500137 0 0) 0x7f0b54004f10 con 0x7f0b7409dce0 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b74123150 con 0x7f0b7409dce0 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706642176 0 0) 0x7f0b7409d340 con 0x7f0b740a1190 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): my global_id is 18079 2024-09-18T11:49:56.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b54004f10 con 0x7f0b740a1190 2024-09-18T11:49:56.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0b5c001d10 con 0x7f0b7409dce0 2024-09-18T11:49:56.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:49:56.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): dump: 2024-09-18T11:49:56.672 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.672 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.672 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.672 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.672 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3829209331 0 0) 0x7f0b74123150 con 0x7f0b7409dce0 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.112:6790/0 conn(0x7f0b7406eaa0 legacy=0x7f0b7411eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6789/0 conn(0x7f0b740a1190 legacy=0x7f0b7414a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient: _finish_hunting 0 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 monclient: found mon.a 2024-09-18T11:49:56.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7411fcf0 con 0x7f0b7409dce0 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 10 monclient: _finish_auth 0 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b6affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 5 monclient: authenticate success, global_id 94161 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 10 monclient: _renew_subs 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b74120ee0 con 0x7f0b7409dce0 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 10 client.94161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 10 monclient: _renew_subs 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b741209d0 con 0x7f0b7409dce0 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.657+0000 7f0b78b5e700 1 librados: init done 2024-09-18T11:49:56.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b5c0023b0 con 0x7f0b7409dce0 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0b5c002740 con 0x7f0b7409dce0 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b7159a700 4 set_mon_vals no callback set 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 10 monclient: dump: 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.675 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.676 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.676 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b78b5e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0b58006590 con 0x7f0b58002840 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0b5c0028a0 con 0x7f0b7409dce0 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(261..261 src has 1..261) v4 ==== 5324+0+0 (unknown 3957991064 0 0) 0x7f0b5c043e80 con 0x7f0b7409dce0 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 10 client.94161.objecter ms_dispatch 0x7f0b74000fd0 osd_map(261..261 src has 1..261) v4 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 3 client.94161.objecter handle_osd_map got epochs [261,261] > 0 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 3 client.94161.objecter handle_osd_map decoding full epoch 261 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 20 client.94161.objecter dump_active .. 0 homeless 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b72d9d700 10 monclient: get_auth_request con 0x7f0b54038500 auth_method 0 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 10 client.94161.objecter ms_handle_connect 0x7f0b58002840 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b6affd700 10 client.94161.objecter resend_mon_ops 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.661+0000 7f0b7259c700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6789/0 conn(0x7f0b58002840 legacy=0x7f0b58004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098530876 0 0) 0x7f0b58006590 con 0x7f0b58002840 2024-09-18T11:49:56.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 monclient(hunting): my global_id is 18082 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b74123150 con 0x7f0b58002840 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 client.94161.objecter ms_handle_connect 0x7f0b54038500 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2229846399 0 0) 0x7f0b74123150 con 0x7f0b58002840 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b58006590 con 0x7f0b58002840 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0b64001a90 con 0x7f0b58002840 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 monclient: dump: 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.679 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2587119464 0 0) 0x7f0b58006590 con 0x7f0b58002840 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b5800a140 con 0x7f0b58002840 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f0b64005860 con 0x7f0b58002840 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:49:56.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6789/0 conn(0x7f0b58002840 legacy=0x7f0b58004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.665+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6789/0 conn(0x7f0b58002840 legacy=0x7f0b58004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:56.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.673+0000 7f0b78b5e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:49:56.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.673+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0b58009f10 con 0x7f0b58006900 2024-09-18T11:49:56.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b7259c700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6789/0 conn(0x7f0b58006900 legacy=0x7f0b58006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:49:56.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 client.94161.objecter ms_handle_connect 0x7f0b58006900 2024-09-18T11:49:56.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 client.94161.objecter resend_mon_ops 2024-09-18T11:49:56.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108968995 0 0) 0x7f0b58009f10 con 0x7f0b58006900 2024-09-18T11:49:56.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:49:56.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:49:56.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 monclient(hunting): my global_id is 18085 2024-09-18T11:49:56.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b58006590 con 0x7f0b58006900 2024-09-18T11:49:56.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2176847875 0 0) 0x7f0b58006590 con 0x7f0b58006900 2024-09-18T11:49:56.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b58009f10 con 0x7f0b58006900 2024-09-18T11:49:56.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0b64004c80 con 0x7f0b58006900 2024-09-18T11:49:56.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 monclient: dump: 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3561573652 0 0) 0x7f0b58009f10 con 0x7f0b58006900 2024-09-18T11:49:56.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0b58008760 con 0x7f0b58006900 2024-09-18T11:49:56.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 1468500204) 0x7f0b64005b10 con 0x7f0b58006900 2024-09-18T11:49:56.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:49:56.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:49:56.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6789/0 conn(0x7f0b58006900 legacy=0x7f0b58006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.677+0000 7f0b6affd700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6789/0 conn(0x7f0b58006900 legacy=0x7f0b58006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:49:56.685 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 28, 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:56.686 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:49:56.687 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:49:56.688 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:49:56.689 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:49:56.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.691 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.692 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:49:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:49:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:49:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:49:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:49:56.697 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:49:56.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 10 librados: watch_flush enter 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 10 librados: watch_flush exit 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 20 client.94161.objecter shutdown clearing up homeless session... 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 10 client.94161.objecter successfully canceled tick 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.129:6812/20806 conn(0x7f0b54038500 legacy=0x7f0b5403a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 10 monclient: shutdown 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 >> v1:172.21.15.112:6789/0 conn(0x7f0b7409dce0 legacy=0x7f0b7411f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b72d9d700 1 -- 172.21.15.112:0/708712874 reap_dead start 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 shutdown_connections 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 >> 172.21.15.112:0/708712874 conn(0x7f0b740a4570 msgr2=0x7f0b7409bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 shutdown_connections 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 1 -- 172.21.15.112:0/708712874 wait complete. 2024-09-18T11:49:56.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:49:56.681+0000 7f0b78b5e700 1 librados: shutdown 2024-09-18T11:49:56.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:49:56.700 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T11:49:56.700 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-18T11:49:56.700 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T11:49:56.700 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T11:49:56.700 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T11:49:56.700 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:49:56.715 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T11:49:56.715 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T11:49:56.715 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T11:49:56.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T11:50:11.716 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T11:50:11.716 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T11:50:11.717 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T11:50:11.717 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:50:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.821+0000 7fbdc3fff700 10 monclient: get_monmap_and_config 2024-09-18T11:50:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.821+0000 7fbdc3fff700 10 monclient: build_initial_monmap 2024-09-18T11:50:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.821+0000 7fbdc3fff700 10 monclient: monmap: 2024-09-18T11:50:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:50:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:50:11.827345+0000 2024-09-18T11:50:11.828 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:50:11.827345+0000 2024-09-18T11:50:11.828 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:50:11.828 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:11.828 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:50:11.828 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:50:11.828 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:50:11.828 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:11.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.821+0000 7fbdc3fff700 1 Processor -- start 2024-09-18T11:50:11.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.821+0000 7fbdc3fff700 1 -- start start 2024-09-18T11:50:11.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.821+0000 7fbdc3fff700 10 monclient: init 2024-09-18T11:50:11.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 10 monclient: _reopen_session rank -1 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbdc40a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fbdc412edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fbdc412f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc40981a0 con 0x7fbdc412edc0 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc406c120 con 0x7fbdc40a2040 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc406eef0 con 0x7fbdc412f8a0 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 10 monclient(hunting): _renew_subs 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc3fff700 10 monclient(hunting): authenticate will time out at 1183.668455s 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc37fe700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdc4137230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc27fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbdc412edc0 0x7fbdc412f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33508/0 (socket says 172.21.15.112:33508) 2024-09-18T11:50:11.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc27fc700 1 -- 172.21.15.112:0/3879721872 learned_addr learned my addr 172.21.15.112:0/3879721872 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:50:11.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc27fc700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6789/0 conn(0x7fbdc412edc0 legacy=0x7fbdc412f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:11.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:11.825+0000 7fbdc2ffd700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.129:6789/0 conn(0x7fbdc40a2040 legacy=0x7fbdc4097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:12.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:12.029+0000 7fbdc37fe700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdc4137230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:12.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:12.433+0000 7fbdc37fe700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdc4137230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:13.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:13.237+0000 7fbdc37fe700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdc4137230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 10 monclient(hunting): tick 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 1 monclient(hunting): continuing hunt 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdc4137230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.129:6789/0 conn(0x7fbdc40a2040 legacy=0x7fbdc4097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6789/0 conn(0x7fbdc412edc0 legacy=0x7fbdc412f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 10 monclient(hunting): picked mon.noname-a con 0x7fbdb0003810 addr v1:172.21.15.112:6789/0 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 10 monclient(hunting): picked mon.noname-c con 0x7fbdb0007490 addr v1:172.21.15.129:6789/0 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 10 monclient(hunting): picked mon.noname-b con 0x7fbdc412f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:50:14.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 1 -- 172.21.15.112:0/3879721872 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc4096dd0 con 0x7fbdb0003810 2024-09-18T11:50:14.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 1 -- 172.21.15.112:0/3879721872 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc406eef0 con 0x7fbdb0007490 2024-09-18T11:50:14.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 1 -- 172.21.15.112:0/3879721872 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb0010a20 con 0x7fbdc412f8a0 2024-09-18T11:50:14.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc0ff9700 10 monclient(hunting): _renew_subs 2024-09-18T11:50:14.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc37fe700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdb000d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:14.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc2ffd700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6789/0 conn(0x7fbdb0003810 legacy=0x7fbdb0005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:14.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:14.825+0000 7fbdc27fc700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.129:6789/0 conn(0x7fbdb0007490 legacy=0x7fbdb0009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:15.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:15.029+0000 7fbdc37fe700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdb000d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:15.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:15.433+0000 7fbdc37fe700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdb000d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:16.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:16.237+0000 7fbdc37fe700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdb000d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.241+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322416882 0 0) 0x7fbdc4096dd0 con 0x7fbdb0003810 2024-09-18T11:50:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.241+0000 7fbdc1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.241+0000 7fbdc1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.241+0000 7fbdc1ffb700 10 monclient(hunting): my global_id is 94164 2024-09-18T11:50:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.241+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdac003570 con 0x7fbdb0003810 2024-09-18T11:50:17.258 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.253+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772495843 0 0) 0x7fbdc406eef0 con 0x7fbdb0007490 2024-09-18T11:50:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.253+0000 7fbdc1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.253+0000 7fbdc1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.253+0000 7fbdc1ffb700 10 monclient(hunting): my global_id is 18088 2024-09-18T11:50:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.253+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc4096dd0 con 0x7fbdb0007490 2024-09-18T11:50:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc2ffd700 1 --1- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6789/0 conn(0x7fbdb0003810 0x7fbdb0005c50 :-1 s=OPENED pgs=391 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7fbdb40018a0 mon_map magic: 0 v1 2024-09-18T11:50:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547138239 0 0) 0x7fbdac003570 con 0x7fbdb0003810 2024-09-18T11:50:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdb000d200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.129:6789/0 conn(0x7fbdb0007490 legacy=0x7fbdb0009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 1 monclient: found mon.noname-a 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0011ce0 con 0x7fbdb0003810 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 10 monclient: _finish_auth 0 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc3fff700 5 monclient: authenticate success, global_id 94164 2024-09-18T11:50:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.285+0000 7fbdc3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:50:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbdb40018a0 con 0x7fbdb0003810 2024-09-18T11:50:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc1ffb700 10 monclient: dump: 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:50:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdb4001a80 con 0x7fbdb0003810 2024-09-18T11:50:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:50:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 10 monclient: get_monmap_and_config success 2024-09-18T11:50:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 4 set_mon_vals no callback set 2024-09-18T11:50:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 10 monclient: shutdown 2024-09-18T11:50:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:50:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3879721872 >> v1:172.21.15.112:6789/0 conn(0x7fbdb0003810 legacy=0x7fbdb0005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc1ffb700 1 -- 172.21.15.112:0/3879721872 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbdb4001f00 con 0x7fbdb0003810 2024-09-18T11:50:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc1ffb700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3879721872 shutdown_connections 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3879721872 >> 172.21.15.112:0/3879721872 conn(0x7fbdc4067da0 msgr2=0x7fbdc40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3879721872 shutdown_connections 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.289+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3879721872 wait complete. 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient: build_initial_monmap 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient: monmap: 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:50:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 librados: starting msgr at 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 librados: starting objecter 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 Processor -- start 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 -- start start 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 librados: setting wanted keys 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 librados: calling monclient init 2024-09-18T11:50:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient: init 2024-09-18T11:50:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient: _reopen_session rank -1 2024-09-18T11:50:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:50:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fbdb0003810 addr v1:172.21.15.129:6789/0 2024-09-18T11:50:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fbdb0007490 addr v1:172.21.15.112:6789/0 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbdc412f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc411cbd0 con 0x7fbdb0007490 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc411e680 con 0x7fbdb0003810 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc411e960 con 0x7fbdc412f8a0 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient(hunting): _renew_subs 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc3fff700 10 monclient(hunting): authenticate will time out at 1189.137888s 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc37fe700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdc4149790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc27fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbdb0007490 0x7fbdc412e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33526/0 (socket says 172.21.15.112:33526) 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.293+0000 7fbdc27fc700 1 -- 172.21.15.112:0/3834541718 learned_addr learned my addr 172.21.15.112:0/3834541718 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc27fc700 1 -- 172.21.15.112:0/3834541718 >> v1:172.21.15.112:6789/0 conn(0x7fbdb0007490 legacy=0x7fbdc412e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 client.?.objecter ms_handle_connect 0x7fbdb0007490 2024-09-18T11:50:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:50:17.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc2ffd700 1 -- 172.21.15.112:0/3834541718 >> v1:172.21.15.129:6789/0 conn(0x7fbdb0003810 legacy=0x7fbdc411c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 client.?.objecter ms_handle_connect 0x7fbdb0003810 2024-09-18T11:50:17.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:50:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101999286 0 0) 0x7fbdc411e680 con 0x7fbdb0003810 2024-09-18T11:50:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): my global_id is 18091 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb000a8b0 con 0x7fbdb0003810 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573271181 0 0) 0x7fbdc411cbd0 con 0x7fbdb0007490 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): my global_id is 94167 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc411e680 con 0x7fbdb0007490 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3094620479 0 0) 0x7fbdb000a8b0 con 0x7fbdb0003810 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdc411cbd0 con 0x7fbdb0003810 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbdb4002510 con 0x7fbdb0003810 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): dump: 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 780237273 0 0) 0x7fbdc411cbd0 con 0x7fbdb0003810 2024-09-18T11:50:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 >> v1:172.21.15.112:6790/0 conn(0x7fbdc412f8a0 legacy=0x7fbdc4149790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 >> v1:172.21.15.112:6789/0 conn(0x7fbdb0007490 legacy=0x7fbdc412e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient: _finish_hunting 0 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 monclient: found mon.b 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc411ec40 con 0x7fbdb0003810 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient: _finish_auth 0 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 5 monclient: authenticate success, global_id 18091 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 10 monclient: _renew_subs 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:50:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc412e960 con 0x7fbdb0003810 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 10 client.18091.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 10 monclient: _renew_subs 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc409d1a0 con 0x7fbdb0003810 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 1 librados: init done 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdb4001700 con 0x7fbdb0003810 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbdb4002890 con 0x7fbdb0003810 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc17fa700 4 set_mon_vals no callback set 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:50:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 10 monclient: dump: 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fbdb4002af0 con 0x7fbdb0003810 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:50:17.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:50:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.297+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda40050b0 con 0x7fbdb0003810 2024-09-18T11:50:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.301+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(261..261 src has 1..261) v4 ==== 5324+0+0 (unknown 3957991064 0 0) 0x7fbdb4043d40 con 0x7fbdb0003810 2024-09-18T11:50:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.301+0000 7fbdaaffd700 10 client.18091.objecter ms_dispatch 0x7fbdc4000fd0 osd_map(261..261 src has 1..261) v4 2024-09-18T11:50:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.301+0000 7fbdaaffd700 3 client.18091.objecter handle_osd_map got epochs [261,261] > 0 2024-09-18T11:50:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.301+0000 7fbdaaffd700 3 client.18091.objecter handle_osd_map decoding full epoch 261 2024-09-18T11:50:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.301+0000 7fbdaaffd700 20 client.18091.objecter dump_active .. 0 homeless 2024-09-18T11:50:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.301+0000 7fbdc27fc700 10 monclient: get_auth_request con 0x7fbdb0038f10 auth_method 0 2024-09-18T11:50:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.301+0000 7fbdaaffd700 10 client.18091.objecter ms_handle_connect 0x7fbdb0038f10 2024-09-18T11:50:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.309+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fbdb4014a10 con 0x7fbdb0003810 2024-09-18T11:50:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.309+0000 7fbdaaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:50:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.309+0000 7fbdaaffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:50:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.533+0000 7fbdc3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:50:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.533+0000 7fbdc3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:50:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.533+0000 7fbdc3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:50:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.533+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbda40058e0 con 0x7fbdb0003810 2024-09-18T11:50:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.533+0000 7fbdaaffd700 1 -- 172.21.15.112:0/3834541718 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2519410956) 0x7fbdb4014410 con 0x7fbdb0003810 2024-09-18T11:50:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.533+0000 7fbdaaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:50:17.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.533+0000 7fbdaaffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:50:17.539 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:50:17.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 10 librados: watch_flush enter 2024-09-18T11:50:17.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 10 librados: watch_flush exit 2024-09-18T11:50:17.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 20 client.18091.objecter shutdown clearing up homeless session... 2024-09-18T11:50:17.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 10 client.18091.objecter successfully canceled tick 2024-09-18T11:50:17.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 >> v1:172.21.15.129:6812/20806 conn(0x7fbdb0038f10 legacy=0x7fbdb003b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 10 monclient: shutdown 2024-09-18T11:50:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:50:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 >> v1:172.21.15.129:6789/0 conn(0x7fbdb0003810 legacy=0x7fbdc411c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 shutdown_connections 2024-09-18T11:50:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 >> 172.21.15.112:0/3834541718 conn(0x7fbdc4067da0 msgr2=0x7fbdc4131de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:50:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 shutdown_connections 2024-09-18T11:50:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 1 -- 172.21.15.112:0/3834541718 wait complete. 2024-09-18T11:50:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.537+0000 7fbdc3fff700 1 librados: shutdown 2024-09-18T11:50:17.552 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T11:50:17.552 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T11:50:17.553 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:50:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient: get_monmap_and_config 2024-09-18T11:50:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient: build_initial_monmap 2024-09-18T11:50:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient: monmap: 2024-09-18T11:50:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:50:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:50:17.658243+0000 2024-09-18T11:50:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:50:17.658243+0000 2024-09-18T11:50:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:50:17.659 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.659 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:50:17.659 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:50:17.659 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:50:17.659 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 1 Processor -- start 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 1 -- start start 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient: init 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient: _reopen_session rank -1 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient(hunting): picked mon.noname-c con 0x7f300c0a1230 addr v1:172.21.15.129:6789/0 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient(hunting): picked mon.noname-b con 0x7f300c06e9d0 addr v1:172.21.15.112:6790/0 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient(hunting): picked mon.noname-a con 0x7f300c135300 addr v1:172.21.15.112:6789/0 2024-09-18T11:50:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c0a1600 con 0x7f300c135300 2024-09-18T11:50:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c06eda0 con 0x7f300c0a1230 2024-09-18T11:50:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c06b5a0 con 0x7f300c06e9d0 2024-09-18T11:50:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient(hunting): _renew_subs 2024-09-18T11:50:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3012a60700 10 monclient(hunting): authenticate will time out at 1189.499606s 2024-09-18T11:50:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f30037fe700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f300c06e9d0 legacy=0x7f300c06ae90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3010ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f300c135300 0x7f300c1376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33530/0 (socket says 172.21.15.112:33530) 2024-09-18T11:50:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.653+0000 7f3010ffd700 1 -- 172.21.15.112:0/122944907 learned_addr learned my addr 172.21.15.112:0/122944907 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:50:17.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f3010ffd700 1 -- 172.21.15.112:0/122944907 >> v1:172.21.15.112:6789/0 conn(0x7f300c135300 legacy=0x7f300c1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300bfff700 1 -- 172.21.15.112:0/122944907 >> v1:172.21.15.129:6789/0 conn(0x7f300c0a1230 legacy=0x7f300c06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389465818 0 0) 0x7f300c0a1600 con 0x7f300c135300 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient(hunting): my global_id is 94170 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff8003570 con 0x7f300c135300 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070543666 0 0) 0x7f300c06eda0 con 0x7f300c0a1230 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient(hunting): my global_id is 18094 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c0a1600 con 0x7f300c0a1230 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790929341 0 0) 0x7f2ff8003570 con 0x7f300c135300 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 >> v1:172.21.15.112:6790/0 conn(0x7f300c06e9d0 legacy=0x7f300c06ae90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:50:17.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 >> v1:172.21.15.129:6789/0 conn(0x7f300c0a1230 legacy=0x7f300c06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 monclient: found mon.noname-a 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c09f9e0 con 0x7f300c135300 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: _finish_auth 0 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f3012a60700 5 monclient: authenticate success, global_id 94170 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3004001cc0 con 0x7f300c135300 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f3012a60700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: dump: 2024-09-18T11:50:17.665 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f3012a60700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3004001f50 con 0x7f300c135300 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f3012a60700 10 monclient: get_monmap_and_config success 2024-09-18T11:50:17.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f3012a60700 4 set_mon_vals no callback set 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 1 -- 172.21.15.112:0/122944907 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3004004e10 con 0x7f300c135300 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.657+0000 7f300b7fe700 10 monclient: dump: 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.667 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient: shutdown 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- 172.21.15.112:0/122944907 >> v1:172.21.15.112:6789/0 conn(0x7f300c135300 legacy=0x7f300c1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- 172.21.15.112:0/122944907 shutdown_connections 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- 172.21.15.112:0/122944907 >> 172.21.15.112:0/122944907 conn(0x7f300c0680d0 msgr2=0x7f300c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- 172.21.15.112:0/122944907 shutdown_connections 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- 172.21.15.112:0/122944907 wait complete. 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient: build_initial_monmap 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient: monmap: 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:50:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 librados: starting msgr at 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 librados: starting objecter 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 Processor -- start 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- start start 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 librados: setting wanted keys 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 librados: calling monclient init 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient: init 2024-09-18T11:50:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient: _reopen_session rank -1 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient(hunting): picked mon.noname-b con 0x7f300c06e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient(hunting): picked mon.noname-c con 0x7f300c0a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient(hunting): picked mon.noname-a con 0x7f300c135300 addr v1:172.21.15.112:6789/0 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c118f60 con 0x7f300c135300 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c12efc0 con 0x7f300c06e9d0 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f30037fe700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f300c0a1230 legacy=0x7f300c146a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c1189a0 con 0x7f300c0a1230 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient(hunting): _renew_subs 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3012a60700 10 monclient(hunting): authenticate will time out at 1189.506533s 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3010ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f300c135300 0x7f300c14a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33532/0 (socket says 172.21.15.112:33532) 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.661+0000 7f3010ffd700 1 -- 172.21.15.112:0/1138836309 learned_addr learned my addr 172.21.15.112:0/1138836309 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3010ffd700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6789/0 conn(0x7f300c135300 legacy=0x7f300c14a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 client.?.objecter ms_handle_connect 0x7f300c135300 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 client.?.objecter ms_handle_connect 0x7f300c06e9d0 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f300bfff700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.129:6789/0 conn(0x7f300c06e9d0 legacy=0x7f300c12e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112296429 0 0) 0x7f300c12efc0 con 0x7f300c06e9d0 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient(hunting): my global_id is 18097 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fec004f10 con 0x7f300c06e9d0 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1274987825 0 0) 0x7f2fec004f10 con 0x7f300c06e9d0 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f300c12efc0 con 0x7f300c06e9d0 2024-09-18T11:50:17.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2ffc001c00 con 0x7f300c06e9d0 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient(hunting): dump: 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 690459353 0 0) 0x7f300c12efc0 con 0x7f300c06e9d0 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6790/0 conn(0x7f300c0a1230 legacy=0x7f300c146a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6789/0 conn(0x7f300c135300 legacy=0x7f300c14a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 monclient: found mon.b 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c118c80 con 0x7f300c06e9d0 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient: _finish_auth 0 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 5 monclient: authenticate success, global_id 18097 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 10 monclient: _renew_subs 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:50:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c13ef70 con 0x7f300c06e9d0 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 10 client.18097.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 10 monclient: _renew_subs 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c13e830 con 0x7f300c06e9d0 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 1 librados: init done 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ffc0022b0 con 0x7f300c06e9d0 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2ffc002640 con 0x7f300c06e9d0 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f300affd700 4 set_mon_vals no callback set 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3008ff9700 10 monclient: dump: 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:50:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.665+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ff0006590 con 0x7f2ff0002840 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f2ffc0027a0 con 0x7f300c06e9d0 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(261..261 src has 1..261) v4 ==== 5324+0+0 (unknown 3957991064 0 0) 0x7f2ffc043f30 con 0x7f300c06e9d0 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 client.18097.objecter ms_dispatch 0x7f300c000fd0 osd_map(261..261 src has 1..261) v4 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 3 client.18097.objecter handle_osd_map got epochs [261,261] > 0 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 3 client.18097.objecter handle_osd_map decoding full epoch 261 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 20 client.18097.objecter dump_active .. 0 homeless 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3010ffd700 10 monclient: get_auth_request con 0x7f2fec0380a0 auth_method 0 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f30037fe700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6789/0 conn(0x7f2ff0002840 legacy=0x7f2ff0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 client.18097.objecter ms_handle_connect 0x7f2ff0002840 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 client.18097.objecter resend_mon_ops 2024-09-18T11:50:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698914236 0 0) 0x7f2ff0006590 con 0x7f2ff0002840 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 monclient(hunting): my global_id is 94173 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c12efc0 con 0x7f2ff0002840 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3829120232 0 0) 0x7f300c12efc0 con 0x7f2ff0002840 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ff0006590 con 0x7f2ff0002840 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2ff40042c0 con 0x7f2ff0002840 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 monclient: dump: 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3214395908 0 0) 0x7f2ff0006590 con 0x7f2ff0002840 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ff000a140 con 0x7f2ff0002840 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2ff40065e0 con 0x7f2ff0002840 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.669+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6789/0 conn(0x7f2ff0002840 legacy=0x7f2ff0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.673+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6789/0 conn(0x7f2ff0002840 legacy=0x7f2ff0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:50:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.673+0000 7f3008ff9700 10 client.18097.objecter ms_handle_connect 0x7f2fec0380a0 2024-09-18T11:50:17.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.681+0000 7f3012a60700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:50:17.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.681+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ff0009670 con 0x7f2ff00090e0 2024-09-18T11:50:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 client.18097.objecter ms_handle_connect 0x7f2ff00090e0 2024-09-18T11:50:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 client.18097.objecter resend_mon_ops 2024-09-18T11:50:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f30037fe700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6789/0 conn(0x7f2ff00090e0 legacy=0x7f2ff0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218631069 0 0) 0x7f2ff0009670 con 0x7f2ff00090e0 2024-09-18T11:50:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 monclient(hunting): my global_id is 94176 2024-09-18T11:50:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff0006590 con 0x7f2ff00090e0 2024-09-18T11:50:17.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 145211799 0 0) 0x7f2ff0006590 con 0x7f2ff00090e0 2024-09-18T11:50:17.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ff0009670 con 0x7f2ff00090e0 2024-09-18T11:50:17.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2ff4004aa0 con 0x7f2ff00090e0 2024-09-18T11:50:17.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 monclient: dump: 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3091678459 0 0) 0x7f2ff0009670 con 0x7f2ff00090e0 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2ff0009930 con 0x7f2ff00090e0 2024-09-18T11:50:17.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3822 (unknown 0 0 1620541583) 0x7f2ff4005ee0 con 0x7f2ff00090e0 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6789/0 conn(0x7f2ff00090e0 legacy=0x7f2ff0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3008ff9700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.112:6789/0 conn(0x7f2ff00090e0 legacy=0x7f2ff0008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 32, 2024-09-18T11:50:17.692 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: 1 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:50:17.693 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:50:17.694 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:50:17.695 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:50:17.696 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:50:17.697 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:50:17.698 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:50:17.699 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:50:17.700 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:50:17.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.702 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:50:17.703 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:50:17.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 10 librados: watch_flush enter 2024-09-18T11:50:17.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 10 librados: watch_flush exit 2024-09-18T11:50:17.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 20 client.18097.objecter shutdown clearing up homeless session... 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 10 client.18097.objecter successfully canceled tick 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.129:6812/20806 conn(0x7f2fec0380a0 legacy=0x7f2fec03a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 10 monclient: shutdown 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 >> v1:172.21.15.129:6789/0 conn(0x7f300c06e9d0 legacy=0x7f300c12e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3010ffd700 1 -- 172.21.15.112:0/1138836309 reap_dead start 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 shutdown_connections 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 >> 172.21.15.112:0/1138836309 conn(0x7f300c0680d0 msgr2=0x7f300c134650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 shutdown_connections 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 1 -- 172.21.15.112:0/1138836309 wait complete. 2024-09-18T11:50:17.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.685+0000 7f3012a60700 1 librados: shutdown 2024-09-18T11:50:17.706 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:50:17.706 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T11:50:17.706 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:50:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient: get_monmap_and_config 2024-09-18T11:50:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient: build_initial_monmap 2024-09-18T11:50:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient: monmap: 2024-09-18T11:50:17.816 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:50:17.816606+0000 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:50:17.816606+0000 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:50:17.817 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 1 Processor -- start 2024-09-18T11:50:17.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 1 -- start start 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient: init 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient: _reopen_session rank -1 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient(hunting): picked mon.noname-b con 0x7fad8c0a15f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient(hunting): picked mon.noname-a con 0x7fad8c0a19c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient(hunting): picked mon.noname-c con 0x7fad8c138530 addr v1:172.21.15.129:6789/0 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c0a3250 con 0x7fad8c0a19c0 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c064370 con 0x7fad8c138530 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c06a5a0 con 0x7fad8c0a15f0 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient(hunting): _renew_subs 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad92f5e700 10 monclient(hunting): authenticate will time out at 1189.657455s 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad91f5c700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fad8c0a15f0 legacy=0x7fad8c06c750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad9175b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fad8c0a19c0 0x7fad8c0a1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33550/0 (socket says 172.21.15.112:33550) 2024-09-18T11:50:17.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad9175b700 1 -- 172.21.15.112:0/3708820555 learned_addr learned my addr 172.21.15.112:0/3708820555 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.813+0000 7fad9175b700 1 -- 172.21.15.112:0/3708820555 >> v1:172.21.15.112:6789/0 conn(0x7fad8c0a19c0 legacy=0x7fad8c0a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096328257 0 0) 0x7fad8c0a3250 con 0x7fad8c0a19c0 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient(hunting): my global_id is 94179 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad78003570 con 0x7fad8c0a19c0 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad9275d700 1 -- 172.21.15.112:0/3708820555 >> v1:172.21.15.129:6789/0 conn(0x7fad8c138530 legacy=0x7fad8c13a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052697660 0 0) 0x7fad78003570 con 0x7fad8c0a19c0 2024-09-18T11:50:17.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 >> v1:172.21.15.112:6790/0 conn(0x7fad8c0a15f0 legacy=0x7fad8c06c750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 >> v1:172.21.15.129:6789/0 conn(0x7fad8c138530 legacy=0x7fad8c13a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: _finish_hunting 0 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 monclient: found mon.noname-a 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c06b590 con 0x7fad8c0a19c0 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: _finish_auth 0 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fad88001d80 con 0x7fad8c0a19c0 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: dump: 2024-09-18T11:50:17.823 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 5 monclient: authenticate success, global_id 94179 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad88001f60 con 0x7fad8c0a19c0 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:50:17.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 1 -- 172.21.15.112:0/3708820555 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fad88004e10 con 0x7fad8c0a19c0 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 10 monclient: get_monmap_and_config success 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 4 set_mon_vals no callback set 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad90f5a700 10 monclient: dump: 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 10 monclient: shutdown 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 -- 172.21.15.112:0/3708820555 >> v1:172.21.15.112:6789/0 conn(0x7fad8c0a19c0 legacy=0x7fad8c0a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 -- 172.21.15.112:0/3708820555 shutdown_connections 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 -- 172.21.15.112:0/3708820555 >> 172.21.15.112:0/3708820555 conn(0x7fad8c0a0da0 msgr2=0x7fad8c067680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 -- 172.21.15.112:0/3708820555 shutdown_connections 2024-09-18T11:50:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 -- 172.21.15.112:0/3708820555 wait complete. 2024-09-18T11:50:17.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 10 monclient: build_initial_monmap 2024-09-18T11:50:17.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 10 monclient: monmap: 2024-09-18T11:50:17.830 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:50:17.830 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:50:17.830 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:50:17.830 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:50:17.830 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 librados: starting msgr at 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 librados: starting objecter 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 Processor -- start 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 -- start start 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 librados: setting wanted keys 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 1 librados: calling monclient init 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.817+0000 7fad92f5e700 10 monclient: init 2024-09-18T11:50:17.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient: _reopen_session rank -1 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient(hunting): picked mon.noname-b con 0x7fad8c0a15f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient(hunting): picked mon.noname-a con 0x7fad8c0a19c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient(hunting): picked mon.noname-c con 0x7fad8c138530 addr v1:172.21.15.112:6790/0 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c1362b0 con 0x7fad8c0a19c0 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c06e3d0 con 0x7fad8c0a15f0 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c06f000 con 0x7fad8c138530 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient(hunting): _renew_subs 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient(hunting): authenticate will time out at 1189.664400s 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad9275d700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fad8c138530 legacy=0x7fad8c149c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad9175b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fad8c0a19c0 0x7fad8c146550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33558/0 (socket says 172.21.15.112:33558) 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad9175b700 1 -- 172.21.15.112:0/766808625 learned_addr learned my addr 172.21.15.112:0/766808625 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad9175b700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.112:6789/0 conn(0x7fad8c0a19c0 legacy=0x7fad8c146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 client.?.objecter ms_handle_connect 0x7fad8c0a19c0 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489360253 0 0) 0x7fad8c1362b0 con 0x7fad8c0a19c0 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): my global_id is 94182 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad91f5c700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6789/0 conn(0x7fad8c0a15f0 legacy=0x7fad8c132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad70004f10 con 0x7fad8c0a19c0 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 client.?.objecter ms_handle_connect 0x7fad8c0a15f0 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1094634990 0 0) 0x7fad70004f10 con 0x7fad8c0a19c0 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad8c1362b0 con 0x7fad8c0a19c0 2024-09-18T11:50:17.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fad88001cc0 con 0x7fad8c0a19c0 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): dump: 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:50:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145683408 0 0) 0x7fad8c06e3d0 con 0x7fad8c0a15f0 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient(hunting): my global_id is 18103 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad70004f10 con 0x7fad8c0a15f0 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 228869521 0 0) 0x7fad8c1362b0 con 0x7fad8c0a19c0 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.112:6790/0 conn(0x7fad8c138530 legacy=0x7fad8c149c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6789/0 conn(0x7fad8c0a15f0 legacy=0x7fad8c132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 monclient: found mon.a 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c137860 con 0x7fad8c0a19c0 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient: _finish_auth 0 2024-09-18T11:50:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 5 monclient: authenticate success, global_id 94182 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient: _renew_subs 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8c132850 con 0x7fad8c0a19c0 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 client.94182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient: _renew_subs 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8c135d30 con 0x7fad8c0a19c0 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad92f5e700 1 librados: init done 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad88005340 con 0x7fad8c0a19c0 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fad88005650 con 0x7fad8c0a19c0 2024-09-18T11:50:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad83fff700 4 set_mon_vals no callback set 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.821+0000 7fad81ffb700 10 monclient: dump: 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.837 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad92f5e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fad6c006590 con 0x7fad6c002840 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fad8800fed0 con 0x7fad8c0a19c0 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(261..261 src has 1..261) v4 ==== 5324+0+0 (unknown 3957991064 0 0) 0x7fad88044420 con 0x7fad8c0a19c0 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 10 client.94182.objecter ms_dispatch 0x7fad8c000fd0 osd_map(261..261 src has 1..261) v4 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 3 client.94182.objecter handle_osd_map got epochs [261,261] > 0 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 3 client.94182.objecter handle_osd_map decoding full epoch 261 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 20 client.94182.objecter dump_active .. 0 homeless 2024-09-18T11:50:17.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad9275d700 10 monclient: get_auth_request con 0x7fad70038100 auth_method 0 2024-09-18T11:50:17.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 10 client.94182.objecter ms_handle_connect 0x7fad6c002840 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 10 client.94182.objecter resend_mon_ops 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad91f5c700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6789/0 conn(0x7fad6c002840 legacy=0x7fad6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016022288 0 0) 0x7fad6c006590 con 0x7fad6c002840 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 10 monclient(hunting): my global_id is 18106 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.825+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad8c1362b0 con 0x7fad6c002840 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 10 client.94182.objecter ms_handle_connect 0x7fad70038100 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 735891099 0 0) 0x7fad8c1362b0 con 0x7fad6c002840 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad6c006590 con 0x7fad6c002840 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fad7c0042e0 con 0x7fad6c002840 2024-09-18T11:50:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 10 monclient: dump: 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1280601798 0 0) 0x7fad6c006590 con 0x7fad6c002840 2024-09-18T11:50:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad6c00a140 con 0x7fad6c002840 2024-09-18T11:50:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fad7c006620 con 0x7fad6c002840 2024-09-18T11:50:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:50:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:50:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6789/0 conn(0x7fad6c002840 legacy=0x7fad6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.829+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6789/0 conn(0x7fad6c002840 legacy=0x7fad6c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:50:17.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.837+0000 7fad92f5e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:50:17.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.837+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fad6c007550 con 0x7fad6c006900 2024-09-18T11:50:17.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad91f5c700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6789/0 conn(0x7fad6c006900 legacy=0x7fad6c006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:50:17.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 10 client.94182.objecter ms_handle_connect 0x7fad6c006900 2024-09-18T11:50:17.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 10 client.94182.objecter resend_mon_ops 2024-09-18T11:50:17.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766848035 0 0) 0x7fad6c007550 con 0x7fad6c006900 2024-09-18T11:50:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:50:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:50:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 10 monclient(hunting): my global_id is 18109 2024-09-18T11:50:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad6c006590 con 0x7fad6c006900 2024-09-18T11:50:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 364921470 0 0) 0x7fad6c006590 con 0x7fad6c006900 2024-09-18T11:50:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad6c007550 con 0x7fad6c006900 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fad7c004ac0 con 0x7fad6c006900 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 10 monclient: dump: 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:50:17.848 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:50:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3023958774 0 0) 0x7fad6c007550 con 0x7fad6c006900 2024-09-18T11:50:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.841+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fad6c0087b0 con 0x7fad6c006900 2024-09-18T11:50:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4160 (unknown 0 0 4204296031) 0x7fad7c005ec0 con 0x7fad6c006900 2024-09-18T11:50:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad81ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:50:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad81ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:50:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6789/0 conn(0x7fad6c006900 legacy=0x7fad6c006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad81ffb700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6789/0 conn(0x7fad6c006900 legacy=0x7fad6c006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 32, 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout: 1 2024-09-18T11:50:17.850 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:50:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:50:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:50:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:50:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:50:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:50:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:50:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 6 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:50:17.861 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:50:17.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 10 librados: watch_flush enter 2024-09-18T11:50:17.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 10 librados: watch_flush exit 2024-09-18T11:50:17.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 20 client.94182.objecter shutdown clearing up homeless session... 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 10 client.94182.objecter successfully canceled tick 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.129:6812/20806 conn(0x7fad70038100 legacy=0x7fad7003a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 10 monclient: shutdown 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 >> v1:172.21.15.112:6789/0 conn(0x7fad8c0a19c0 legacy=0x7fad8c146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad9275d700 1 -- 172.21.15.112:0/766808625 reap_dead start 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 shutdown_connections 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 >> 172.21.15.112:0/766808625 conn(0x7fad8c0a0da0 msgr2=0x7fad8c09f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 shutdown_connections 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 1 -- 172.21.15.112:0/766808625 wait complete. 2024-09-18T11:50:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:50:17.845+0000 7fad92f5e700 1 librados: shutdown 2024-09-18T11:50:17.867 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:50:17.867 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T11:50:39.750 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFu3:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:5d165639:::164:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f43765fc:::165:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b4c720e9:::166:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:e694b040:::167:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:afa38db2:::168:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:77ba9f53:::169:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:87495034:::170:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:7c96bf0e:::171:head 2024-09-18T11:50:39.751 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:dbe346cc:::172:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:e943ec24:::173:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f97a9c0c:::174:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:6f26e74d:::175:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:4f95e106:::176:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:0e6f2f8f:::177:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:05db05f1:::178:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:38a78d66:::179:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:d095610b:::180:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:a1a9d709:::181:head 2024-09-18T11:50:39.752 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:1e5d39db:::182:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:f7df4fb9:::183:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:03a7f161:::184:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:ba70721e:::185:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:28e5662d:::186:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:973d52de:::187:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:4303eb1c:::188:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:b990b48e:::189:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:29b8165b:::190:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:3547f197:::191:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:7e260936:::192:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:1abec7b1:::193:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:10fdda93:::194:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:15817eea:::195:head 2024-09-18T11:50:39.753 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:770bab57:::196:head 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:ed9e13e7:::197:head 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:71471a8f:::198:head 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 189:10fb1d02:::199:head 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (9114 ms) 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,0 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first is 1726660175 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,0 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,0 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,0 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,0 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,0 2024-09-18T11:50:39.754 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,1726660180,1726660181,0 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,1726660180,1726660181,0 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,1726660180,1726660181,0 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,1726660180,1726660181,1726660183,1726660184,0 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,1726660180,1726660181,1726660183,1726660184,0 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660175,1726660177,1726660178,1726660180,1726660181,1726660183,1726660184,0 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660178,1726660180,1726660181,1726660183,1726660184,1726660186,1726660187,0 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first now 1726660178, trimmed 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21106 ms) 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: foo0 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (34731 ms) 2024-09-18T11:50:39.755 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-18T11:51:01.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ llPP2 2024-09-18T11:51:01.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (24040 ms) 2024-09-18T11:51:01.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-18T11:51:01.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6472 ms) 2024-09-18T11:51:01.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-18T11:51:01.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7266 ms) 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6761 ms) 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6982 ms) 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7095 ms) 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6949 ms) 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (159000 ms total) 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (337936 ms total) 2024-09-18T11:51:01.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-18T11:51:47.868 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T11:51:47.868 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T11:51:47.869 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T11:51:47.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T11:51:47.873 INFO:tasks.ceph.mon.c:Started 2024-09-18T11:51:47.873 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T11:51:47.873 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T11:51:47.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T11:52:02.874 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T11:52:02.875 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T11:52:02.875 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:52:02.875 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:52:02.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.986+0000 7f0c83a2f700 10 monclient: get_monmap_and_config 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.986+0000 7f0c83a2f700 10 monclient: build_initial_monmap 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.986+0000 7f0c83a2f700 10 monclient: monmap: 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:02.992064+0000 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:02.992064+0000 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:02.992 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:02.993 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:02.993 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 1 Processor -- start 2024-09-18T11:52:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 1 -- start start 2024-09-18T11:52:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 10 monclient: init 2024-09-18T11:52:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:52:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 10 monclient(hunting): picked mon.noname-a con 0x7f0c7c0a0570 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 10 monclient(hunting): picked mon.noname-b con 0x7f0c7c0a1050 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 10 monclient(hunting): picked mon.noname-c con 0x7f0c7c097a50 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:02.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7c097070 con 0x7f0c7c0a0570 2024-09-18T11:52:02.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7c096d90 con 0x7f0c7c097a50 2024-09-18T11:52:02.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7c097e50 con 0x7f0c7c0a1050 2024-09-18T11:52:02.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:02.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c83a2f700 10 monclient(hunting): authenticate will time out at 1294.833551s 2024-09-18T11:52:02.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c80fca700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0c7c0a1050 0x7f0c7c137780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44538/0 (socket says 172.21.15.112:44538) 2024-09-18T11:52:02.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c80fca700 1 -- 172.21.15.112:0/3260906988 learned_addr learned my addr 172.21.15.112:0/3260906988 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:02.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.990+0000 7f0c80fca700 1 -- 172.21.15.112:0/3260906988 >> v1:172.21.15.112:6790/0 conn(0x7f0c7c0a1050 legacy=0x7f0c7c137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:02.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c817cb700 1 -- 172.21.15.112:0/3260906988 >> v1:172.21.15.112:6789/0 conn(0x7f0c7c0a0570 legacy=0x7f0c7c0a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:02.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c81fcc700 1 -- 172.21.15.112:0/3260906988 >> v1:172.21.15.129:6789/0 conn(0x7f0c7c097a50 legacy=0x7f0c7c13aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:02.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125642659 0 0) 0x7f0c7c097070 con 0x7f0c7c0a0570 2024-09-18T11:52:02.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:02.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:02.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): my global_id is 94221 2024-09-18T11:52:02.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c68003570 con 0x7f0c7c0a0570 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436777439 0 0) 0x7f0c7c097e50 con 0x7f0c7c0a1050 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): my global_id is 104099 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c7c097070 con 0x7f0c7c0a1050 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982276082 0 0) 0x7f0c7c096d90 con 0x7f0c7c097a50 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient(hunting): my global_id is 18145 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c7c097e50 con 0x7f0c7c097a50 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174256194 0 0) 0x7f0c68003570 con 0x7f0c7c0a0570 2024-09-18T11:52:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 >> v1:172.21.15.112:6790/0 conn(0x7f0c7c0a1050 legacy=0x7f0c7c137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 >> v1:172.21.15.129:6789/0 conn(0x7f0c7c097a50 legacy=0x7f0c7c13aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 monclient: found mon.noname-a 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c0a2050 con 0x7f0c7c0a0570 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c78001dd0 con 0x7f0c7c0a0570 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: dump: 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 5 monclient: authenticate success, global_id 94221 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c78001fb0 con 0x7f0c7c0a0570 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 1 -- 172.21.15.112:0/3260906988 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c78002420 con 0x7f0c7c0a0570 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 4 set_mon_vals no callback set 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c73fff700 10 monclient: dump: 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient: shutdown 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- 172.21.15.112:0/3260906988 >> v1:172.21.15.112:6789/0 conn(0x7f0c7c0a0570 legacy=0x7f0c7c0a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- 172.21.15.112:0/3260906988 shutdown_connections 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- 172.21.15.112:0/3260906988 >> 172.21.15.112:0/3260906988 conn(0x7f0c7c067680 msgr2=0x7f0c7c067a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- 172.21.15.112:0/3260906988 shutdown_connections 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- 172.21.15.112:0/3260906988 wait complete. 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient: monmap: 2024-09-18T11:52:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 librados: starting msgr at 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 librados: starting objecter 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 Processor -- start 2024-09-18T11:52:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- start start 2024-09-18T11:52:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 librados: setting wanted keys 2024-09-18T11:52:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 librados: calling monclient init 2024-09-18T11:52:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient: init 2024-09-18T11:52:03.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:52:03.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient(hunting): picked mon.noname-b con 0x7f0c7c097a50 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient(hunting): picked mon.noname-a con 0x7f0c7c0a0570 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient(hunting): picked mon.noname-c con 0x7f0c7c0a1050 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7c0a2390 con 0x7f0c7c0a0570 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7c135360 con 0x7f0c7c097a50 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7c121780 con 0x7f0c7c0a1050 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c83a2f700 10 monclient(hunting): authenticate will time out at 1294.839093s 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c80fca700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0c7c0a0570 0x7f0c7c146a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45372/0 (socket says 172.21.15.112:45372) 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c81fcc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0c7c0a1050 0x7f0c7c14a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44540/0 (socket says 172.21.15.112:44540) 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.994+0000 7f0c80fca700 1 -- 172.21.15.112:0/8840778 learned_addr learned my addr 172.21.15.112:0/8840778 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c81fcc700 1 -- 172.21.15.112:0/8840778 >> v1:172.21.15.112:6790/0 conn(0x7f0c7c0a1050 legacy=0x7f0c7c14a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 client.?.objecter ms_handle_connect 0x7f0c7c0a1050 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 client.?.objecter ms_handle_connect 0x7f0c7c0a0570 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c80fca700 1 -- 172.21.15.112:0/8840778 >> v1:172.21.15.112:6789/0 conn(0x7f0c7c0a0570 legacy=0x7f0c7c146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28084543 0 0) 0x7f0c7c121780 con 0x7f0c7c0a1050 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): my global_id is 104102 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c60004f10 con 0x7f0c7c0a1050 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667091093 0 0) 0x7f0c7c0a2390 con 0x7f0c7c0a0570 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): my global_id is 94224 2024-09-18T11:52:03.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c7c121780 con 0x7f0c7c0a0570 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3843284427 0 0) 0x7f0c60004f10 con 0x7f0c7c0a1050 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c7c0a2390 con 0x7f0c7c0a1050 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c74001ba0 con 0x7f0c7c0a1050 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): dump: 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.012 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393010271 0 0) 0x7f0c7c121780 con 0x7f0c7c0a0570 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c60004f10 con 0x7f0c7c0a0570 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c6c001c20 con 0x7f0c7c0a0570 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient(hunting): dump: 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3900231558 0 0) 0x7f0c7c0a2390 con 0x7f0c7c0a1050 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 >> v1:172.21.15.129:6789/0 conn(0x7f0c7c097a50 legacy=0x7f0c7c134380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 >> v1:172.21.15.112:6789/0 conn(0x7f0c7c0a0570 legacy=0x7f0c7c146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 monclient: found mon.c 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:03.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c11f9e0 con 0x7f0c7c0a1050 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 5 monclient: authenticate success, global_id 104102 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 10 monclient: _renew_subs 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7c1214a0 con 0x7f0c7c0a1050 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 10 client.104102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 10 monclient: _renew_subs 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7c134a90 con 0x7f0c7c0a1050 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 1 librados: init done 2024-09-18T11:52:03.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c740021b0 con 0x7f0c7c0a1050 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0c740025c0 con 0x7f0c7c0a1050 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c737fe700 4 set_mon_vals no callback set 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 monclient: dump: 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.016 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0c740027a0 con 0x7f0c7c0a1050 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5c0050b0 con 0x7f0c7c0a1050 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(340..340 src has 1..340) v4 ==== 5393+0+0 (unknown 1637002363 0 0) 0x7f0c74043e80 con 0x7f0c7c0a1050 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 10 client.104102.objecter ms_dispatch 0x7f0c7c000fd0 osd_map(340..340 src has 1..340) v4 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 3 client.104102.objecter handle_osd_map got epochs [340,340] > 0 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 3 client.104102.objecter handle_osd_map decoding full epoch 340 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c717fa700 20 client.104102.objecter dump_active .. 0 homeless 2024-09-18T11:52:03.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:02.998+0000 7f0c817cb700 10 monclient: get_auth_request con 0x7f0c60038200 auth_method 0 2024-09-18T11:52:03.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.002+0000 7f0c717fa700 10 client.104102.objecter ms_handle_connect 0x7f0c60038200 2024-09-18T11:52:03.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.006+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f0c74014a50 con 0x7f0c7c0a1050 2024-09-18T11:52:03.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.006+0000 7f0c717fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.006+0000 7f0c717fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.226+0000 7f0c83a2f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:52:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.226+0000 7f0c83a2f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:52:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.226+0000 7f0c83a2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:03.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.226+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0c5c0058e0 con 0x7f0c7c0a1050 2024-09-18T11:52:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.226+0000 7f0c717fa700 1 -- 172.21.15.112:0/8840778 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2303870875) 0x7f0c74014450 con 0x7f0c7c0a1050 2024-09-18T11:52:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.226+0000 7f0c717fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:52:03.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.226+0000 7f0c717fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:03.230 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 10 librados: watch_flush enter 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 10 librados: watch_flush exit 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 20 client.104102.objecter shutdown clearing up homeless session... 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 10 client.104102.objecter successfully canceled tick 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 >> v1:172.21.15.129:6812/20806 conn(0x7f0c60038200 legacy=0x7f0c6003a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 10 monclient: shutdown 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 >> v1:172.21.15.112:6790/0 conn(0x7f0c7c0a1050 legacy=0x7f0c7c14a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 shutdown_connections 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 >> 172.21.15.112:0/8840778 conn(0x7f0c7c067680 msgr2=0x7f0c7c0a4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 shutdown_connections 2024-09-18T11:52:03.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 1 -- 172.21.15.112:0/8840778 wait complete. 2024-09-18T11:52:03.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.230+0000 7f0c83a2f700 1 librados: shutdown 2024-09-18T11:52:03.244 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:52:03.244 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:52:03.244 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient: get_monmap_and_config 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient: monmap: 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:03.346272+0000 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:03.346272+0000 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.346 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:03.347 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:03.347 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 1 Processor -- start 2024-09-18T11:52:03.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 1 -- start start 2024-09-18T11:52:03.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient: init 2024-09-18T11:52:03.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:52:03.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient(hunting): picked mon.noname-c con 0x7fb9a40a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient(hunting): picked mon.noname-b con 0x7fb9a4098730 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient(hunting): picked mon.noname-a con 0x7fb9a4138310 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a4097af0 con 0x7fb9a4138310 2024-09-18T11:52:03.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a4097780 con 0x7fb9a40a2040 2024-09-18T11:52:03.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a409dea0 con 0x7fb9a4098730 2024-09-18T11:52:03.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a99b8700 10 monclient(hunting): authenticate will time out at 1295.187031s 2024-09-18T11:52:03.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a3fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb9a4098730 0x7fb9a409d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44550/0 (socket says 172.21.15.112:44550) 2024-09-18T11:52:03.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.342+0000 7fb9a3fff700 1 -- 172.21.15.112:0/2052304 learned_addr learned my addr 172.21.15.112:0/2052304 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a3fff700 1 -- 172.21.15.112:0/2052304 >> v1:172.21.15.112:6790/0 conn(0x7fb9a4098730 legacy=0x7fb9a409d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a91b7700 1 -- 172.21.15.112:0/2052304 >> v1:172.21.15.112:6789/0 conn(0x7fb9a4138310 legacy=0x7fb9a413a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a89b6700 1 -- 172.21.15.112:0/2052304 >> v1:172.21.15.129:6789/0 conn(0x7fb9a40a2040 legacy=0x7fb9a409d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667573510 0 0) 0x7fb9a409dea0 con 0x7fb9a4098730 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): my global_id is 104105 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb98c003570 con 0x7fb9a4098730 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842608036 0 0) 0x7fb9a4097af0 con 0x7fb9a4138310 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): my global_id is 94227 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a409dea0 con 0x7fb9a4138310 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774229436 0 0) 0x7fb9a4097780 con 0x7fb9a40a2040 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient(hunting): my global_id is 18148 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4097af0 con 0x7fb9a40a2040 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3396462832 0 0) 0x7fb98c003570 con 0x7fb9a4098730 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 >> v1:172.21.15.129:6789/0 conn(0x7fb9a40a2040 legacy=0x7fb9a409d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 >> v1:172.21.15.112:6789/0 conn(0x7fb9a4138310 legacy=0x7fb9a413a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 monclient: found mon.noname-b 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a409e640 con 0x7fb9a4098730 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb998001940 con 0x7fb9a4098730 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: dump: 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.354 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 5 monclient: authenticate success, global_id 104105 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb998001bd0 con 0x7fb9a4098730 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 1 -- 172.21.15.112:0/2052304 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb998004a90 con 0x7fb9a4098730 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a37fe700 10 monclient: dump: 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.355 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 4 set_mon_vals no callback set 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 10 monclient: shutdown 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2052304 >> v1:172.21.15.112:6790/0 conn(0x7fb9a4098730 legacy=0x7fb9a409d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2052304 shutdown_connections 2024-09-18T11:52:03.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.346+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2052304 >> 172.21.15.112:0/2052304 conn(0x7fb9a4067da0 msgr2=0x7fb9a40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2052304 shutdown_connections 2024-09-18T11:52:03.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2052304 wait complete. 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient: monmap: 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.362 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 librados: starting msgr at 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 librados: starting objecter 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 Processor -- start 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 -- start start 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 librados: setting wanted keys 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 librados: calling monclient init 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient: init 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:52:03.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient(hunting): picked mon.noname-a con 0x7fb9a4098730 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient(hunting): picked mon.noname-c con 0x7fb9a40a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient(hunting): picked mon.noname-b con 0x7fb9a4138310 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a4096a60 con 0x7fb9a4098730 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a89b6700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb9a4098730 0x7fb9a41412c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45388/0 (socket says 172.21.15.112:45388) 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a4130470 con 0x7fb9a4138310 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a89b6700 1 -- 172.21.15.112:0/2677241336 learned_addr learned my addr 172.21.15.112:0/2677241336 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a4130750 con 0x7fb9a40a2040 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.350+0000 7fb9a99b8700 10 monclient(hunting): authenticate will time out at 1295.195901s 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a89b6700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6789/0 conn(0x7fb9a4098730 legacy=0x7fb9a41412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 client.?.objecter ms_handle_connect 0x7fb9a4098730 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 client.?.objecter ms_handle_connect 0x7fb9a40a2040 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a3fff700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6790/0 conn(0x7fb9a40a2040 legacy=0x7fb9a41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200281200 0 0) 0x7fb9a4096a60 con 0x7fb9a4098730 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): my global_id is 94230 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990004f10 con 0x7fb9a4098730 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634773848 0 0) 0x7fb9a4130750 con 0x7fb9a40a2040 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): my global_id is 104108 2024-09-18T11:52:03.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4096a60 con 0x7fb9a40a2040 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 client.?.objecter ms_handle_connect 0x7fb9a4138310 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a91b7700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.129:6789/0 conn(0x7fb9a4138310 legacy=0x7fb9a4149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1473153878 0 0) 0x7fb990004f10 con 0x7fb9a4098730 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a4130750 con 0x7fb9a4098730 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb994001d80 con 0x7fb9a4098730 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): dump: 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.366 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 185979105 0 0) 0x7fb9a4096a60 con 0x7fb9a40a2040 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb990004f10 con 0x7fb9a40a2040 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb998001360 con 0x7fb9a40a2040 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): dump: 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311526870 0 0) 0x7fb9a4130470 con 0x7fb9a4138310 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient(hunting): my global_id is 18151 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4096a60 con 0x7fb9a4138310 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2068001532 0 0) 0x7fb9a4130750 con 0x7fb9a4098730 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6790/0 conn(0x7fb9a40a2040 legacy=0x7fb9a41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.129:6789/0 conn(0x7fb9a4138310 legacy=0x7fb9a4149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 monclient: found mon.a 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a412f580 con 0x7fb9a4098730 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 5 monclient: authenticate success, global_id 94230 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 10 monclient: _renew_subs 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a41419d0 con 0x7fb9a4098730 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 10 client.94230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 10 monclient: _renew_subs 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a4137e80 con 0x7fb9a4098730 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a99b8700 1 librados: init done 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb994002330 con 0x7fb9a4098730 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb994002640 con 0x7fb9a4098730 2024-09-18T11:52:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a2ffd700 4 set_mon_vals no callback set 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.354+0000 7fb9a0ff9700 10 monclient: dump: 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.371 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a99b8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb984006590 con 0x7fb984002840 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb9940028d0 con 0x7fb9a4098730 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(340..340 src has 1..340) v4 ==== 5393+0+0 (unknown 1637002363 0 0) 0x7fb994043fc0 con 0x7fb9a4098730 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 client.94230.objecter ms_dispatch 0x7fb9a4000fd0 osd_map(340..340 src has 1..340) v4 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 3 client.94230.objecter handle_osd_map got epochs [340,340] > 0 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 3 client.94230.objecter handle_osd_map decoding full epoch 340 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 20 client.94230.objecter dump_active .. 0 homeless 2024-09-18T11:52:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a91b7700 10 monclient: get_auth_request con 0x7fb990038460 auth_method 0 2024-09-18T11:52:03.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 client.94230.objecter ms_handle_connect 0x7fb984002840 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 client.94230.objecter resend_mon_ops 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a3fff700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6789/0 conn(0x7fb984002840 legacy=0x7fb984004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210845530 0 0) 0x7fb984006590 con 0x7fb984002840 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 monclient(hunting): my global_id is 94233 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4130750 con 0x7fb984002840 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 260652494 0 0) 0x7fb9a4130750 con 0x7fb984002840 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb984006590 con 0x7fb984002840 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb998004e20 con 0x7fb984002840 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.358+0000 7fb9a0ff9700 10 monclient: dump: 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.362+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1374066627 0 0) 0x7fb984006590 con 0x7fb984002840 2024-09-18T11:52:03.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.362+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb98400a140 con 0x7fb984002840 2024-09-18T11:52:03.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.362+0000 7fb9a0ff9700 10 client.94230.objecter ms_handle_connect 0x7fb990038460 2024-09-18T11:52:03.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.362+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fb998006fc0 con 0x7fb984002840 2024-09-18T11:52:03.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.362+0000 7fb9a0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.362+0000 7fb9a0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:03.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.362+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6789/0 conn(0x7fb984002840 legacy=0x7fb984004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.362+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6789/0 conn(0x7fb984002840 legacy=0x7fb984004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:03.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a99b8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:52:03.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb984009670 con 0x7fb9840090e0 2024-09-18T11:52:03.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a3fff700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6789/0 conn(0x7fb9840090e0 legacy=0x7fb984008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a0ff9700 10 client.94230.objecter ms_handle_connect 0x7fb9840090e0 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a0ff9700 10 client.94230.objecter resend_mon_ops 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604216254 0 0) 0x7fb984009670 con 0x7fb9840090e0 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a0ff9700 10 monclient(hunting): my global_id is 94236 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.370+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb984006590 con 0x7fb9840090e0 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4218620953 0 0) 0x7fb984006590 con 0x7fb9840090e0 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb984009670 con 0x7fb9840090e0 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb998005450 con 0x7fb9840090e0 2024-09-18T11:52:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 10 monclient: dump: 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1074365483 0 0) 0x7fb984009670 con 0x7fb9840090e0 2024-09-18T11:52:03.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb984009930 con 0x7fb9840090e0 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4003 (unknown 0 0 586502224) 0x7fb9980068d0 con 0x7fb9840090e0 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6789/0 conn(0x7fb9840090e0 legacy=0x7fb984008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a0ff9700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6789/0 conn(0x7fb9840090e0 legacy=0x7fb984008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:03.380 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 36, 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.381 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:52:03.383 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:52:03.384 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.385 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.386 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:52:03.387 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:52:03.387 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:52:03.388 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:03.389 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.390 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:52:03.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.392 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:52:03.393 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:52:03.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 10 librados: watch_flush enter 2024-09-18T11:52:03.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 10 librados: watch_flush exit 2024-09-18T11:52:03.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 20 client.94230.objecter shutdown clearing up homeless session... 2024-09-18T11:52:03.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 10 client.94230.objecter successfully canceled tick 2024-09-18T11:52:03.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.129:6812/20806 conn(0x7fb990038460 legacy=0x7fb99003a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 10 monclient: shutdown 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 >> v1:172.21.15.112:6789/0 conn(0x7fb9a4098730 legacy=0x7fb9a41412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a91b7700 1 -- 172.21.15.112:0/2677241336 reap_dead start 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a91b7700 1 -- 172.21.15.112:0/2677241336 reap_dead start 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 shutdown_connections 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 >> 172.21.15.112:0/2677241336 conn(0x7fb9a4067da0 msgr2=0x7fb9a413a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 shutdown_connections 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 1 -- 172.21.15.112:0/2677241336 wait complete. 2024-09-18T11:52:03.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.374+0000 7fb9a99b8700 1 librados: shutdown 2024-09-18T11:52:03.396 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:52:03.396 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:52:03.396 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:52:03.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.502+0000 7f9e44829700 10 monclient: get_monmap_and_config 2024-09-18T11:52:03.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.502+0000 7f9e44829700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.502+0000 7f9e44829700 10 monclient: monmap: 2024-09-18T11:52:03.507 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.507 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.507 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:03.507609+0000 2024-09-18T11:52:03.507 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:03.507609+0000 2024-09-18T11:52:03.508 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.508 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.508 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.508 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:03.508 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:03.508 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 1 Processor -- start 2024-09-18T11:52:03.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 1 -- start start 2024-09-18T11:52:03.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 10 monclient: init 2024-09-18T11:52:03.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 10 monclient(hunting): picked mon.noname-b con 0x7f9e400a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 10 monclient(hunting): picked mon.noname-a con 0x7f9e400a0e60 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 10 monclient(hunting): picked mon.noname-c con 0x7f9e40097970 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e40097550 con 0x7f9e400a0e60 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e40096cb0 con 0x7f9e40097970 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e400a1440 con 0x7f9e400a0380 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e44829700 10 monclient(hunting): authenticate will time out at 1295.349403s 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e3f7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f9e400a0380 0x7f9e400a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44572/0 (socket says 172.21.15.112:44572) 2024-09-18T11:52:03.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e3f7fe700 1 -- 172.21.15.112:0/2852057174 learned_addr learned my addr 172.21.15.112:0/2852057174 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e3f7fe700 1 -- 172.21.15.112:0/2852057174 >> v1:172.21.15.112:6790/0 conn(0x7f9e400a0380 legacy=0x7f9e400a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.506+0000 7f9e3effd700 1 -- 172.21.15.112:0/2852057174 >> v1:172.21.15.112:6789/0 conn(0x7f9e400a0e60 legacy=0x7f9e40137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630243707 0 0) 0x7f9e400a1440 con 0x7f9e400a0380 2024-09-18T11:52:03.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient(hunting): my global_id is 104111 2024-09-18T11:52:03.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e28003570 con 0x7f9e400a0380 2024-09-18T11:52:03.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369433548 0 0) 0x7f9e40097550 con 0x7f9e400a0e60 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient(hunting): my global_id is 94239 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e400a1440 con 0x7f9e400a0e60 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3ffff700 1 -- 172.21.15.112:0/2852057174 >> v1:172.21.15.129:6789/0 conn(0x7f9e40097970 legacy=0x7f9e4013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491998854 0 0) 0x7f9e28003570 con 0x7f9e400a0380 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 >> v1:172.21.15.129:6789/0 conn(0x7f9e40097970 legacy=0x7f9e4013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 >> v1:172.21.15.112:6789/0 conn(0x7f9e400a0e60 legacy=0x7f9e40137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 monclient: found mon.noname-b 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:52:03.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e400a1de0 con 0x7f9e400a0380 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9e30001a00 con 0x7f9e400a0380 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: dump: 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.517 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 5 monclient: authenticate success, global_id 104111 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e30001be0 con 0x7f9e400a0380 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 4 set_mon_vals no callback set 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 1 -- 172.21.15.112:0/2852057174 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9e30004a90 con 0x7f9e400a0380 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e3e7fc700 10 monclient: dump: 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 10 monclient: shutdown 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 -- 172.21.15.112:0/2852057174 >> v1:172.21.15.112:6790/0 conn(0x7f9e400a0380 legacy=0x7f9e400a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 -- 172.21.15.112:0/2852057174 shutdown_connections 2024-09-18T11:52:03.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 -- 172.21.15.112:0/2852057174 >> 172.21.15.112:0/2852057174 conn(0x7f9e40068100 msgr2=0x7f9e400a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 -- 172.21.15.112:0/2852057174 shutdown_connections 2024-09-18T11:52:03.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 -- 172.21.15.112:0/2852057174 wait complete. 2024-09-18T11:52:03.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 10 monclient: monmap: 2024-09-18T11:52:03.527 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.527 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:03.527 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:03.527 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:03.527 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 librados: starting msgr at 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 librados: starting objecter 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 Processor -- start 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.510+0000 7f9e44829700 1 -- start start 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 1 librados: setting wanted keys 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 1 librados: calling monclient init 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 10 monclient: init 2024-09-18T11:52:03.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 10 monclient(hunting): picked mon.noname-c con 0x7f9e40097970 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 10 monclient(hunting): picked mon.noname-a con 0x7f9e400a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 10 monclient(hunting): picked mon.noname-b con 0x7f9e400a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e400a2050 con 0x7f9e400a0380 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e40098520 con 0x7f9e400a0e60 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e4011c8d0 con 0x7f9e40097970 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e44829700 10 monclient(hunting): authenticate will time out at 1295.357148s 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e3f7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f9e40097970 0x7f9e4013e970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44576/0 (socket says 172.21.15.112:44576) 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e3f7fe700 1 -- 172.21.15.112:0/2937146239 learned_addr learned my addr 172.21.15.112:0/2937146239 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e3effd700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6789/0 conn(0x7f9e400a0380 legacy=0x7f9e401467f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 client.?.objecter ms_handle_connect 0x7f9e400a0380 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 client.?.objecter ms_handle_connect 0x7f9e40097970 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e3f7fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6790/0 conn(0x7f9e40097970 legacy=0x7f9e4013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e3ffff700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.129:6789/0 conn(0x7f9e400a0e60 legacy=0x7f9e40149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 client.?.objecter ms_handle_connect 0x7f9e400a0e60 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151984562 0 0) 0x7f9e400a2050 con 0x7f9e400a0380 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 monclient(hunting): my global_id is 94242 2024-09-18T11:52:03.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2c004f10 con 0x7f9e400a0380 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336080149 0 0) 0x7f9e4011c8d0 con 0x7f9e40097970 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 10 monclient(hunting): my global_id is 104114 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.514+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e400a2050 con 0x7f9e40097970 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726994422 0 0) 0x7f9e40098520 con 0x7f9e400a0e60 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): my global_id is 18154 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e4011c8d0 con 0x7f9e400a0e60 2024-09-18T11:52:03.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 910539640 0 0) 0x7f9e2c004f10 con 0x7f9e400a0380 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e40098520 con 0x7f9e400a0380 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9e34001c00 con 0x7f9e400a0380 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): dump: 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.532 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2039744725 0 0) 0x7f9e400a2050 con 0x7f9e40097970 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e2c004f10 con 0x7f9e40097970 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9e300017a0 con 0x7f9e40097970 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient(hunting): dump: 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.533 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3666125301 0 0) 0x7f9e40098520 con 0x7f9e400a0380 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6790/0 conn(0x7f9e40097970 legacy=0x7f9e4013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.129:6789/0 conn(0x7f9e400a0e60 legacy=0x7f9e40149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 monclient: found mon.a 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4013b1a0 con 0x7f9e400a0380 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 5 monclient: authenticate success, global_id 94242 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 10 monclient: _renew_subs 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4013c730 con 0x7f9e400a0380 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 10 client.94242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:03.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 10 monclient: _renew_subs 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e40135ac0 con 0x7f9e400a0380 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 1 librados: init done 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e340021b0 con 0x7f9e400a0380 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9e34002500 con 0x7f9e400a0380 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e3d7fa700 4 set_mon_vals no callback set 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 monclient: dump: 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.536 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9e20006590 con 0x7f9e20002840 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f9e34002660 con 0x7f9e400a0380 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(340..340 src has 1..340) v4 ==== 5393+0+0 (unknown 1637002363 0 0) 0x7f9e34043dc0 con 0x7f9e400a0380 2024-09-18T11:52:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 10 client.94242.objecter ms_dispatch 0x7f9e40000fd0 osd_map(340..340 src has 1..340) v4 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 3 client.94242.objecter handle_osd_map got epochs [340,340] > 0 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 3 client.94242.objecter handle_osd_map decoding full epoch 340 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.518+0000 7f9e277fe700 20 client.94242.objecter dump_active .. 0 homeless 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e3f7fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6790/0 conn(0x7f9e20002840 legacy=0x7f9e20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 client.94242.objecter ms_handle_connect 0x7f9e20002840 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 client.94242.objecter resend_mon_ops 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455021768 0 0) 0x7f9e20006590 con 0x7f9e20002840 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 monclient(hunting): my global_id is 104117 2024-09-18T11:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e40098520 con 0x7f9e20002840 2024-09-18T11:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e3ffff700 10 monclient: get_auth_request con 0x7f9e2c038570 auth_method 0 2024-09-18T11:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2719890200 0 0) 0x7f9e40098520 con 0x7f9e20002840 2024-09-18T11:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e20006590 con 0x7f9e20002840 2024-09-18T11:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9e300051b0 con 0x7f9e20002840 2024-09-18T11:52:03.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 monclient: dump: 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.540 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 982522735 0 0) 0x7f9e20006590 con 0x7f9e20002840 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e2000a140 con 0x7f9e20002840 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f9e30005380 con 0x7f9e20002840 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6790/0 conn(0x7f9e20002840 legacy=0x7f9e20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6790/0 conn(0x7f9e20002840 legacy=0x7f9e20004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:03.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.522+0000 7f9e277fe700 10 client.94242.objecter ms_handle_connect 0x7f9e2c038570 2024-09-18T11:52:03.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e44829700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:52:03.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9e20007550 con 0x7f9e20006900 2024-09-18T11:52:03.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e277fe700 10 client.94242.objecter ms_handle_connect 0x7f9e20006900 2024-09-18T11:52:03.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e277fe700 10 client.94242.objecter resend_mon_ops 2024-09-18T11:52:03.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e3f7fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6790/0 conn(0x7f9e20006900 legacy=0x7f9e20006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046940697 0 0) 0x7f9e20007550 con 0x7f9e20006900 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e277fe700 10 monclient(hunting): my global_id is 104120 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.534+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e20006590 con 0x7f9e20006900 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2736858014 0 0) 0x7f9e20006590 con 0x7f9e20006900 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e20007550 con 0x7f9e20006900 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9e30005a00 con 0x7f9e20006900 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 10 monclient: dump: 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.544 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 924779601 0 0) 0x7f9e20007550 con 0x7f9e20006900 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9e200087b0 con 0x7f9e20006900 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 2504194154) 0x7f9e30006f00 con 0x7f9e20006900 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:52:03.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6790/0 conn(0x7f9e20006900 legacy=0x7f9e20006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e277fe700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6790/0 conn(0x7f9e20006900 legacy=0x7f9e20006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 36, 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:52:03.546 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.547 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.548 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:03.549 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.550 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.552 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.554 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.556 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:03.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.557 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.557 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:52:03.557 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 10 librados: watch_flush enter 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 10 librados: watch_flush exit 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 20 client.94242.objecter shutdown clearing up homeless session... 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 10 client.94242.objecter successfully canceled tick 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.129:6812/20806 conn(0x7f9e2c038570 legacy=0x7f9e2c03aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 10 monclient: shutdown 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e3ffff700 1 -- 172.21.15.112:0/2937146239 reap_dead start 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 >> v1:172.21.15.112:6789/0 conn(0x7f9e400a0380 legacy=0x7f9e401467f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 shutdown_connections 2024-09-18T11:52:03.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 >> 172.21.15.112:0/2937146239 conn(0x7f9e40068100 msgr2=0x7f9e400a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 shutdown_connections 2024-09-18T11:52:03.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 1 -- 172.21.15.112:0/2937146239 wait complete. 2024-09-18T11:52:03.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.538+0000 7f9e44829700 1 librados: shutdown 2024-09-18T11:52:03.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:52:03.560 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:52:03.560 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:52:03.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient: get_monmap_and_config 2024-09-18T11:52:03.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient: monmap: 2024-09-18T11:52:03.669 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.669 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.669 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:03.669522+0000 2024-09-18T11:52:03.669 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:03.669522+0000 2024-09-18T11:52:03.670 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.670 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.670 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.670 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:03.670 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:03.670 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 1 Processor -- start 2024-09-18T11:52:03.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 1 -- start start 2024-09-18T11:52:03.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient: init 2024-09-18T11:52:03.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient(hunting): picked mon.noname-a con 0x7f57e006e470 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient(hunting): picked mon.noname-b con 0x7f57e00a2c50 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient(hunting): picked mon.noname-c con 0x7f57e0136410 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e00a30c0 con 0x7f57e006e470 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e006d760 con 0x7f57e0136410 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e00a2620 con 0x7f57e00a2c50 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e771e700 10 monclient(hunting): authenticate will time out at 1295.511144s 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e4cb9700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f57e00a2c50 0x7f57e00a1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44600/0 (socket says 172.21.15.112:44600) 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e54ba700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f57e006e470 0x7f57e00a1800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45440/0 (socket says 172.21.15.112:45440) 2024-09-18T11:52:03.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.666+0000 7f57e4cb9700 1 -- 172.21.15.112:0/3396788545 learned_addr learned my addr 172.21.15.112:0/3396788545 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e54ba700 1 -- 172.21.15.112:0/3396788545 >> v1:172.21.15.112:6789/0 conn(0x7f57e006e470 legacy=0x7f57e00a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e4cb9700 1 -- 172.21.15.112:0/3396788545 >> v1:172.21.15.112:6790/0 conn(0x7f57e00a2c50 legacy=0x7f57e00a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e5cbb700 1 -- 172.21.15.112:0/3396788545 >> v1:172.21.15.129:6789/0 conn(0x7f57e0136410 legacy=0x7f57e01387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496175723 0 0) 0x7f57e00a30c0 con 0x7f57e006e470 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): my global_id is 94245 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57cc003570 con 0x7f57e006e470 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594175507 0 0) 0x7f57e00a2620 con 0x7f57e00a2c50 2024-09-18T11:52:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): my global_id is 104123 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e00a30c0 con 0x7f57e00a2c50 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649061929 0 0) 0x7f57e006d760 con 0x7f57e0136410 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient(hunting): my global_id is 18157 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e00a2620 con 0x7f57e0136410 2024-09-18T11:52:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 254722852 0 0) 0x7f57cc003570 con 0x7f57e006e470 2024-09-18T11:52:03.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 >> v1:172.21.15.112:6790/0 conn(0x7f57e00a2c50 legacy=0x7f57e00a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 >> v1:172.21.15.129:6789/0 conn(0x7f57e0136410 legacy=0x7f57e01387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 monclient: found mon.noname-a 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e00a32a0 con 0x7f57e006e470 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 5 monclient: authenticate success, global_id 94245 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f57dc001940 con 0x7f57e006e470 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: dump: 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:03.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57dc001bd0 con 0x7f57e006e470 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 4 set_mon_vals no callback set 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 1 -- 172.21.15.112:0/3396788545 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f57dc0049d0 con 0x7f57e006e470 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57d7fff700 10 monclient: dump: 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.682 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 10 monclient: shutdown 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 1 -- 172.21.15.112:0/3396788545 >> v1:172.21.15.112:6789/0 conn(0x7f57e006e470 legacy=0x7f57e00a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 1 -- 172.21.15.112:0/3396788545 shutdown_connections 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 1 -- 172.21.15.112:0/3396788545 >> 172.21.15.112:0/3396788545 conn(0x7f57e0068110 msgr2=0x7f57e00a3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 1 -- 172.21.15.112:0/3396788545 shutdown_connections 2024-09-18T11:52:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.670+0000 7f57e771e700 1 -- 172.21.15.112:0/3396788545 wait complete. 2024-09-18T11:52:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient: monmap: 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:03.689 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 librados: starting msgr at 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 librados: starting objecter 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 Processor -- start 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 -- start start 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 librados: setting wanted keys 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 librados: calling monclient init 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient: init 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient(hunting): picked mon.noname-b con 0x7f57e006e470 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient(hunting): picked mon.noname-a con 0x7f57e00a2c50 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient(hunting): picked mon.noname-c con 0x7f57e0136410 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e006ebb0 con 0x7f57e00a2c50 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e012f270 con 0x7f57e006e470 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e009da80 con 0x7f57e0136410 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e771e700 10 monclient(hunting): authenticate will time out at 1295.518031s 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e4cb9700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f57e00a2c50 0x7f57e01469c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45450/0 (socket says 172.21.15.112:45450) 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.674+0000 7f57e4cb9700 1 -- 172.21.15.112:0/3139433150 learned_addr learned my addr 172.21.15.112:0/3139433150 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e4cb9700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.112:6789/0 conn(0x7f57e00a2c50 legacy=0x7f57e01469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 client.?.objecter ms_handle_connect 0x7f57e00a2c50 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 client.?.objecter ms_handle_connect 0x7f57e0136410 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e5cbb700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.112:6790/0 conn(0x7f57e0136410 legacy=0x7f57e014a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285921820 0 0) 0x7f57e006ebb0 con 0x7f57e00a2c50 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): my global_id is 94248 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c4004f10 con 0x7f57e00a2c50 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397917787 0 0) 0x7f57e009da80 con 0x7f57e0136410 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): my global_id is 104126 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e006ebb0 con 0x7f57e0136410 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908235234 0 0) 0x7f57c4004f10 con 0x7f57e00a2c50 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57e009da80 con 0x7f57e00a2c50 2024-09-18T11:52:03.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f57d0004830 con 0x7f57e00a2c50 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): dump: 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.693 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4155602354 0 0) 0x7f57e006ebb0 con 0x7f57e0136410 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57c4004f10 con 0x7f57e0136410 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f57d8001f40 con 0x7f57e0136410 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e54ba700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6789/0 conn(0x7f57e006e470 legacy=0x7f57e009d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient(hunting): dump: 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 client.?.objecter ms_handle_connect 0x7f57e006e470 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 914149805 0 0) 0x7f57e009da80 con 0x7f57e00a2c50 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.112:6790/0 conn(0x7f57e0136410 legacy=0x7f57e014a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6789/0 conn(0x7f57e006e470 legacy=0x7f57e009d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 monclient: found mon.a 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e009b5a0 con 0x7f57e00a2c50 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 5 monclient: authenticate success, global_id 94248 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 10 monclient: _renew_subs 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e009b880 con 0x7f57e00a2c50 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 10 client.94248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 10 monclient: _renew_subs 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e009a830 con 0x7f57e00a2c50 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 1 librados: init done 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f57e013f910 con 0x7f57e013f540 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57d0004c40 con 0x7f57e00a2c50 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f57d0004f50 con 0x7f57e00a2c50 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 monclient: dump: 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.697 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f57d000f7d0 con 0x7f57e00a2c50 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d6ffd700 4 set_mon_vals no callback set 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(340..340 src has 1..340) v4 ==== 5393+0+0 (unknown 1637002363 0 0) 0x7f57d0043f90 con 0x7f57e00a2c50 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 10 client.94248.objecter ms_dispatch 0x7f57e0000fd0 osd_map(340..340 src has 1..340) v4 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 3 client.94248.objecter handle_osd_map got epochs [340,340] > 0 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 3 client.94248.objecter handle_osd_map decoding full epoch 340 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.678+0000 7f57d57fa700 20 client.94248.objecter dump_active .. 0 homeless 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57e5cbb700 10 monclient: get_auth_request con 0x7f57c40383d0 auth_method 0 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 client.94248.objecter ms_handle_connect 0x7f57e013f540 2024-09-18T11:52:03.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 client.94248.objecter resend_mon_ops 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57e54ba700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6789/0 conn(0x7f57e013f540 legacy=0x7f57e0133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 client.94248.objecter ms_handle_connect 0x7f57c40383d0 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026831071 0 0) 0x7f57e013f910 con 0x7f57e013f540 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 monclient(hunting): my global_id is 18163 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e009da80 con 0x7f57e013f540 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2859277870 0 0) 0x7f57e009da80 con 0x7f57e013f540 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57e013f910 con 0x7f57e013f540 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f57dc001d40 con 0x7f57e013f540 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:52:03.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.682+0000 7f57d57fa700 10 monclient: dump: 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.686+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3461658577 0 0) 0x7f57e013f910 con 0x7f57e013f540 2024-09-18T11:52:03.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.686+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57e009bbf0 con 0x7f57e013f540 2024-09-18T11:52:03.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.686+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f57dc004830 con 0x7f57e013f540 2024-09-18T11:52:03.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.686+0000 7f57d57fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.686+0000 7f57d57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:03.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.686+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6789/0 conn(0x7f57e013f540 legacy=0x7f57e0133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.686+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6789/0 conn(0x7f57e013f540 legacy=0x7f57e0133890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:03.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.698+0000 7f57e771e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:52:03.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.698+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f57e005f2e0 con 0x7f57e002ad10 2024-09-18T11:52:03.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57e54ba700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6789/0 conn(0x7f57e002ad10 legacy=0x7f57e0133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 client.94248.objecter ms_handle_connect 0x7f57e002ad10 2024-09-18T11:52:03.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 client.94248.objecter resend_mon_ops 2024-09-18T11:52:03.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842333108 0 0) 0x7f57e005f2e0 con 0x7f57e002ad10 2024-09-18T11:52:03.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 monclient(hunting): my global_id is 18166 2024-09-18T11:52:03.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e013f910 con 0x7f57e002ad10 2024-09-18T11:52:03.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 922157538 0 0) 0x7f57e013f910 con 0x7f57e002ad10 2024-09-18T11:52:03.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57e005f2e0 con 0x7f57e002ad10 2024-09-18T11:52:03.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f57dc005390 con 0x7f57e002ad10 2024-09-18T11:52:03.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:52:03.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 monclient: dump: 2024-09-18T11:52:03.706 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.706 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 902691005 0 0) 0x7f57e005f2e0 con 0x7f57e002ad10 2024-09-18T11:52:03.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f57e0063680 con 0x7f57e002ad10 2024-09-18T11:52:03.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 1783002961) 0x7f57dc005560 con 0x7f57e002ad10 2024-09-18T11:52:03.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:52:03.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:03.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6789/0 conn(0x7f57e002ad10 legacy=0x7f57e0133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57d57fa700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6789/0 conn(0x7f57e002ad10 legacy=0x7f57e0133fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 36, 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:52:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:52:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:52:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:03.715 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.716 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.718 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:52:03.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:03.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:03.721 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:52:03.721 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57e771e700 10 librados: watch_flush enter 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57e771e700 10 librados: watch_flush exit 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57e771e700 20 client.94248.objecter shutdown clearing up homeless session... 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.702+0000 7f57e771e700 10 client.94248.objecter successfully canceled tick 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.129:6812/20806 conn(0x7f57c40383d0 legacy=0x7f57c403a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 10 monclient: shutdown 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 >> v1:172.21.15.112:6789/0 conn(0x7f57e00a2c50 legacy=0x7f57e01469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e5cbb700 1 -- 172.21.15.112:0/3139433150 reap_dead start 2024-09-18T11:52:03.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e5cbb700 1 -- 172.21.15.112:0/3139433150 reap_dead start 2024-09-18T11:52:03.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 shutdown_connections 2024-09-18T11:52:03.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 >> 172.21.15.112:0/3139433150 conn(0x7f57e0068110 msgr2=0x7f57e00a1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 shutdown_connections 2024-09-18T11:52:03.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 1 -- 172.21.15.112:0/3139433150 wait complete. 2024-09-18T11:52:03.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.706+0000 7f57e771e700 1 librados: shutdown 2024-09-18T11:52:03.724 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:52:03.724 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T11:52:03.724 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T11:52:03.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient: get_monmap_and_config 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient: monmap: 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:03.837979+0000 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:03.837979+0000 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:03.838 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 1 Processor -- start 2024-09-18T11:52:03.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 1 -- start start 2024-09-18T11:52:03.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient: init 2024-09-18T11:52:03.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:52:03.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient(hunting): picked mon.noname-b con 0x7f8828067680 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient(hunting): picked mon.noname-c con 0x7f88280986a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient(hunting): picked mon.noname-a con 0x7f88280961d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882806a240 con 0x7f88280961d0 2024-09-18T11:52:03.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882806fca0 con 0x7f88280986a0 2024-09-18T11:52:03.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8828067a50 con 0x7f8828067680 2024-09-18T11:52:03.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f882d7e1700 10 monclient(hunting): authenticate will time out at 1295.679373s 2024-09-18T11:52:03.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f8826ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8828067680 0x7f8828097f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44618/0 (socket says 172.21.15.112:44618) 2024-09-18T11:52:03.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.834+0000 7f8826ffd700 1 -- 172.21.15.112:0/1475080340 learned_addr learned my addr 172.21.15.112:0/1475080340 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8826ffd700 1 -- 172.21.15.112:0/1475080340 >> v1:172.21.15.112:6790/0 conn(0x7f8828067680 legacy=0x7f8828097f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f88277fe700 1 -- 172.21.15.112:0/1475080340 >> v1:172.21.15.112:6789/0 conn(0x7f88280961d0 legacy=0x7f882813ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f88267fc700 1 -- 172.21.15.112:0/1475080340 >> v1:172.21.15.129:6789/0 conn(0x7f88280986a0 legacy=0x7f88281376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262556195 0 0) 0x7f8828067a50 con 0x7f8828067680 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): my global_id is 104129 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f880c003570 con 0x7f8828067680 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581776218 0 0) 0x7f882806a240 con 0x7f88280961d0 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): my global_id is 94251 2024-09-18T11:52:03.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8828067a50 con 0x7f88280961d0 2024-09-18T11:52:03.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657392094 0 0) 0x7f882806fca0 con 0x7f88280986a0 2024-09-18T11:52:03.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient(hunting): my global_id is 18169 2024-09-18T11:52:03.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f882806a240 con 0x7f88280986a0 2024-09-18T11:52:03.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1631288560 0 0) 0x7f8828067a50 con 0x7f88280961d0 2024-09-18T11:52:03.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 >> v1:172.21.15.112:6790/0 conn(0x7f8828067680 legacy=0x7f8828097f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 >> v1:172.21.15.129:6789/0 conn(0x7f88280986a0 legacy=0x7f88281376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 monclient: found mon.noname-a 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8828133150 con 0x7f88280961d0 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f882d7e1700 5 monclient: authenticate success, global_id 94251 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8820001a30 con 0x7f88280961d0 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: dump: 2024-09-18T11:52:03.850 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f882d7e1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8820001c10 con 0x7f88280961d0 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f882d7e1700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 1 -- 172.21.15.112:0/1475080340 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f88200020a0 con 0x7f88280961d0 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f882d7e1700 4 set_mon_vals no callback set 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.838+0000 7f8825ffb700 10 monclient: dump: 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient: shutdown 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- 172.21.15.112:0/1475080340 >> v1:172.21.15.112:6789/0 conn(0x7f88280961d0 legacy=0x7f882813ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- 172.21.15.112:0/1475080340 shutdown_connections 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- 172.21.15.112:0/1475080340 >> 172.21.15.112:0/1475080340 conn(0x7f88280a2190 msgr2=0x7f88280a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- 172.21.15.112:0/1475080340 shutdown_connections 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- 172.21.15.112:0/1475080340 wait complete. 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient: build_initial_monmap 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient: monmap: 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:03.853 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 librados: starting msgr at 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 librados: starting objecter 2024-09-18T11:52:03.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 Processor -- start 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- start start 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 librados: setting wanted keys 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 librados: calling monclient init 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient: init 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient(hunting): picked mon.noname-c con 0x7f8828067680 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient(hunting): picked mon.noname-b con 0x7f88280961d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient(hunting): picked mon.noname-a con 0x7f88280986a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:03.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882812df20 con 0x7f88280986a0 2024-09-18T11:52:03.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8828130020 con 0x7f88280961d0 2024-09-18T11:52:03.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8828141da0 con 0x7f8828067680 2024-09-18T11:52:03.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:03.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f882d7e1700 10 monclient(hunting): authenticate will time out at 1295.686474s 2024-09-18T11:52:03.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f8826ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8828067680 0x7f8828141550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44628/0 (socket says 172.21.15.112:44628) 2024-09-18T11:52:03.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.842+0000 7f8826ffd700 1 -- 172.21.15.112:0/553193119 learned_addr learned my addr 172.21.15.112:0/553193119 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:03.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f8826ffd700 1 -- 172.21.15.112:0/553193119 >> v1:172.21.15.112:6790/0 conn(0x7f8828067680 legacy=0x7f8828141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 client.?.objecter ms_handle_connect 0x7f8828067680 2024-09-18T11:52:03.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 client.?.objecter ms_handle_connect 0x7f88280986a0 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f88277fe700 1 -- 172.21.15.112:0/553193119 >> v1:172.21.15.112:6789/0 conn(0x7f88280986a0 legacy=0x7f88281498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719874015 0 0) 0x7f8828141da0 con 0x7f8828067680 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): my global_id is 104132 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f88267fc700 1 -- 172.21.15.112:0/553193119 >> v1:172.21.15.129:6789/0 conn(0x7f88280961d0 legacy=0x7f8828131550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8810004f10 con 0x7f8828067680 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 client.?.objecter ms_handle_connect 0x7f88280961d0 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419668871 0 0) 0x7f882812df20 con 0x7f88280986a0 2024-09-18T11:52:03.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): my global_id is 94254 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8828141da0 con 0x7f88280986a0 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492509652 0 0) 0x7f8828130020 con 0x7f88280961d0 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): my global_id is 18172 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f882812df20 con 0x7f88280961d0 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736612913 0 0) 0x7f8828141da0 con 0x7f88280986a0 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8828130020 con 0x7f88280986a0 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2411227030 0 0) 0x7f8810004f10 con 0x7f8828067680 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8828141da0 con 0x7f8828067680 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8820001d80 con 0x7f88280986a0 2024-09-18T11:52:03.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): dump: 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.863 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:03.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f88180019b0 con 0x7f8828067680 2024-09-18T11:52:03.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:03.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient(hunting): dump: 2024-09-18T11:52:03.864 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.864 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.864 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3499465193 0 0) 0x7f8828130020 con 0x7f88280986a0 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 >> v1:172.21.15.112:6790/0 conn(0x7f8828067680 legacy=0x7f8828141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 >> v1:172.21.15.129:6789/0 conn(0x7f88280961d0 legacy=0x7f8828131550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient: _finish_hunting 0 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 monclient: found mon.a 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882812ea10 con 0x7f88280986a0 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient: _finish_auth 0 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 5 monclient: authenticate success, global_id 94254 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 10 monclient: _renew_subs 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882812d570 con 0x7f88280986a0 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 10 client.94254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 10 monclient: _renew_subs 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8828131e00 con 0x7f88280986a0 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 1 librados: init done 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8820001970 con 0x7f88280986a0 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f88200027f0 con 0x7f88280986a0 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f88257fa700 4 set_mon_vals no callback set 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 10 monclient: dump: 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88040050b0 con 0x7f88280986a0 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.846+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8820002950 con 0x7f88280986a0 2024-09-18T11:52:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.850+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(340..340 src has 1..340) v4 ==== 5393+0+0 (unknown 1637002363 0 0) 0x7f8820043ea0 con 0x7f88280986a0 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.850+0000 7f881effd700 10 client.94254.objecter ms_dispatch 0x7f8828000fd0 osd_map(340..340 src has 1..340) v4 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.850+0000 7f881effd700 3 client.94254.objecter handle_osd_map got epochs [340,340] > 0 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.850+0000 7f881effd700 3 client.94254.objecter handle_osd_map decoding full epoch 340 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.850+0000 7f881effd700 20 client.94254.objecter dump_active .. 0 homeless 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.850+0000 7f8826ffd700 10 monclient: get_auth_request con 0x7f8810038260 auth_method 0 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.850+0000 7f881effd700 10 client.94254.objecter ms_handle_connect 0x7f8810038260 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.858+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f8820014b30 con 0x7f88280986a0 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.858+0000 7f881effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:03.858+0000 7f881effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:04.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.086+0000 7f882d7e1700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T11:52:04.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.086+0000 7f882d7e1700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T11:52:04.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.086+0000 7f882d7e1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:04.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.086+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8804005c30 con 0x7f88280986a0 2024-09-18T11:52:04.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f881effd700 1 -- 172.21.15.112:0/553193119 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f8820014530 con 0x7f88280986a0 2024-09-18T11:52:04.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f881effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T11:52:04.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f881effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:04.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=10,kv=8,logm=63} crc {auth=986767716,config=3140514009,health=1890207509,kv=1310296871,logm=2463784252}) 2024-09-18T11:52:04.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=3350828067,config=3140514009,health=92814587,kv=1310296871,logm=1131328607}) 2024-09-18T11:52:04.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=10,kv=8,logm=63} crc {auth=986767716,config=3140514009,health=1890207509,kv=1310296871,logm=2463784252}) 2024-09-18T11:52:04.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=6,logm=64} crc {auth=4140103221,config=3140514009,health=92814587,kv=4275179712,logm=1907998785}) 2024-09-18T11:52:04.095 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.095 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197451703}) 2024-09-18T11:52:04.095 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912907662}) 2024-09-18T11:52:04.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f882d7e1700 10 librados: watch_flush enter 2024-09-18T11:52:04.096 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.096 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197451703}) 2024-09-18T11:52:04.096 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635396151}) 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f882d7e1700 10 librados: watch_flush exit 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f882d7e1700 20 client.94254.objecter shutdown clearing up homeless session... 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f882d7e1700 10 client.94254.objecter successfully canceled tick 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 >> v1:172.21.15.129:6812/20806 conn(0x7f8810038260 legacy=0x7f881003a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f882d7e1700 10 monclient: shutdown 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f882d7e1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.090+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 >> v1:172.21.15.112:6789/0 conn(0x7f88280986a0 legacy=0x7f88281498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 shutdown_connections 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 >> 172.21.15.112:0/553193119 conn(0x7f88280a2190 msgr2=0x7f8828096e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:04.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 shutdown_connections 2024-09-18T11:52:04.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f882d7e1700 1 -- 172.21.15.112:0/553193119 wait complete. 2024-09-18T11:52:04.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f882d7e1700 1 librados: shutdown 2024-09-18T11:52:04.098 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.099 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344836015}) 2024-09-18T11:52:04.099 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4167796601}) 2024-09-18T11:52:04.099 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.099 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344836015}) 2024-09-18T11:52:04.099 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.094+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1685234943}) 2024-09-18T11:52:04.102 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.098+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.102 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.098+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786682796}) 2024-09-18T11:52:04.102 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.098+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1825371314}) 2024-09-18T11:52:04.102 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.098+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.102 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.098+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786682796}) 2024-09-18T11:52:04.102 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.098+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1225282727}) 2024-09-18T11:52:04.104 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.104 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440075867}) 2024-09-18T11:52:04.104 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116822778}) 2024-09-18T11:52:04.104 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.104 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440075867}) 2024-09-18T11:52:04.104 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=224569971}) 2024-09-18T11:52:04.105 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.105 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062547562}) 2024-09-18T11:52:04.106 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4115776847}) 2024-09-18T11:52:04.106 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.106 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062547562}) 2024-09-18T11:52:04.106 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.102+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1026740998}) 2024-09-18T11:52:04.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T11:52:04.118 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.114+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.118 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.114+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955254728}) 2024-09-18T11:52:04.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.114+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1472984802}) 2024-09-18T11:52:04.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.114+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.114+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955254728}) 2024-09-18T11:52:04.119 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.114+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426357508}) 2024-09-18T11:52:04.120 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.120 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=672248302}) 2024-09-18T11:52:04.120 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1505110885}) 2024-09-18T11:52:04.120 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.120 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=672248302}) 2024-09-18T11:52:04.120 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095788857}) 2024-09-18T11:52:04.122 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.122 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273224209}) 2024-09-18T11:52:04.122 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1274797869}) 2024-09-18T11:52:04.122 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.122 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273224209}) 2024-09-18T11:52:04.122 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897111598}) 2024-09-18T11:52:04.123 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.124 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255914444}) 2024-09-18T11:52:04.124 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4031496032}) 2024-09-18T11:52:04.124 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.124 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255914444}) 2024-09-18T11:52:04.124 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.118+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453487060}) 2024-09-18T11:52:04.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745109334}) 2024-09-18T11:52:04.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469682342}) 2024-09-18T11:52:04.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.125 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745109334}) 2024-09-18T11:52:04.126 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1701375104}) 2024-09-18T11:52:04.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202688062}) 2024-09-18T11:52:04.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=289955909}) 2024-09-18T11:52:04.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202688062}) 2024-09-18T11:52:04.127 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.122+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2277705149}) 2024-09-18T11:52:04.128 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.128 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669621186}) 2024-09-18T11:52:04.128 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1074782851}) 2024-09-18T11:52:04.128 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.128 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669621186}) 2024-09-18T11:52:04.128 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301343070}) 2024-09-18T11:52:04.130 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.130 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1718989799}) 2024-09-18T11:52:04.130 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=746183492}) 2024-09-18T11:52:04.130 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.130 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1718989799}) 2024-09-18T11:52:04.130 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3123341327}) 2024-09-18T11:52:04.131 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.132 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798307115}) 2024-09-18T11:52:04.132 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699046217}) 2024-09-18T11:52:04.132 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.132 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798307115}) 2024-09-18T11:52:04.132 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.126+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1070292177}) 2024-09-18T11:52:04.133 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.133 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163813436}) 2024-09-18T11:52:04.133 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1687855513}) 2024-09-18T11:52:04.133 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.133 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163813436}) 2024-09-18T11:52:04.133 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3466422512}) 2024-09-18T11:52:04.134 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.134 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105266023}) 2024-09-18T11:52:04.134 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=918162093}) 2024-09-18T11:52:04.134 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.135 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105266023}) 2024-09-18T11:52:04.135 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532508092}) 2024-09-18T11:52:04.136 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.136 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299561684}) 2024-09-18T11:52:04.136 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427369936}) 2024-09-18T11:52:04.136 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.136 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299561684}) 2024-09-18T11:52:04.136 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.130+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3258096552}) 2024-09-18T11:52:04.137 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.137 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767115736}) 2024-09-18T11:52:04.137 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1538220342}) 2024-09-18T11:52:04.137 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.137 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767115736}) 2024-09-18T11:52:04.137 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3372606721}) 2024-09-18T11:52:04.138 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.138 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070368805}) 2024-09-18T11:52:04.138 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1646575736}) 2024-09-18T11:52:04.138 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.138 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070368805}) 2024-09-18T11:52:04.138 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788342812}) 2024-09-18T11:52:04.139 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.139 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637407709}) 2024-09-18T11:52:04.139 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667448608}) 2024-09-18T11:52:04.139 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.140 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637407709}) 2024-09-18T11:52:04.140 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.134+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1390233106}) 2024-09-18T11:52:04.141 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.141 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330460554}) 2024-09-18T11:52:04.141 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3245419956}) 2024-09-18T11:52:04.141 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.141 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330460554}) 2024-09-18T11:52:04.141 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076057875}) 2024-09-18T11:52:04.142 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.142 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250918422}) 2024-09-18T11:52:04.142 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=566027989}) 2024-09-18T11:52:04.142 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.142 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250918422}) 2024-09-18T11:52:04.142 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805342172}) 2024-09-18T11:52:04.143 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.144 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4042814671}) 2024-09-18T11:52:04.144 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1198996452}) 2024-09-18T11:52:04.144 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.144 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4042814671}) 2024-09-18T11:52:04.144 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.138+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1679442527}) 2024-09-18T11:52:04.145 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.145 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891398888}) 2024-09-18T11:52:04.145 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1304741500}) 2024-09-18T11:52:04.145 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.145 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891398888}) 2024-09-18T11:52:04.145 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608342477}) 2024-09-18T11:52:04.146 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.146 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410280488}) 2024-09-18T11:52:04.146 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775762874}) 2024-09-18T11:52:04.147 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.147 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410280488}) 2024-09-18T11:52:04.147 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1572179839}) 2024-09-18T11:52:04.148 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.148 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301663479}) 2024-09-18T11:52:04.148 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4288630259}) 2024-09-18T11:52:04.148 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.148 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301663479}) 2024-09-18T11:52:04.148 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.142+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=356048324}) 2024-09-18T11:52:04.149 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.149 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543536056}) 2024-09-18T11:52:04.149 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2830749939}) 2024-09-18T11:52:04.149 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.149 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543536056}) 2024-09-18T11:52:04.149 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753138736}) 2024-09-18T11:52:04.150 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.150 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864417929}) 2024-09-18T11:52:04.150 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=630273458}) 2024-09-18T11:52:04.150 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.150 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864417929}) 2024-09-18T11:52:04.150 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1863030133}) 2024-09-18T11:52:04.151 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.151 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277882041}) 2024-09-18T11:52:04.151 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1120044324}) 2024-09-18T11:52:04.151 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.151 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277882041}) 2024-09-18T11:52:04.152 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.146+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149036035}) 2024-09-18T11:52:04.152 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.152 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888188392}) 2024-09-18T11:52:04.152 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2340830371}) 2024-09-18T11:52:04.152 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.152 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888188392}) 2024-09-18T11:52:04.153 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445033967}) 2024-09-18T11:52:04.153 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.153 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669095287}) 2024-09-18T11:52:04.153 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4007017264}) 2024-09-18T11:52:04.153 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.154 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669095287}) 2024-09-18T11:52:04.154 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=880205312}) 2024-09-18T11:52:04.154 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.154 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092192358}) 2024-09-18T11:52:04.154 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2541073564}) 2024-09-18T11:52:04.155 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.155 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092192358}) 2024-09-18T11:52:04.155 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796921329}) 2024-09-18T11:52:04.156 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.156 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822256006}) 2024-09-18T11:52:04.156 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3464588973}) 2024-09-18T11:52:04.156 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.150+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.156 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822256006}) 2024-09-18T11:52:04.157 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3273228102}) 2024-09-18T11:52:04.158 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.158 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3179845746}) 2024-09-18T11:52:04.158 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3812082437}) 2024-09-18T11:52:04.158 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.158 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3179845746}) 2024-09-18T11:52:04.158 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256729669}) 2024-09-18T11:52:04.159 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.159 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=1} crc {logm=3076416300,mdsmap=1531151339,mgr=3797927047,mgr_metadata=454364145,mgrstat=1264896640,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=1703736855,osd_snap=3671887149,osdmap=1214209872}) 2024-09-18T11:52:04.159 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33} crc {logm=930132462,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1264896640,mon_config_key=868353578,monmap=1725997317,osd_metadata=3039517214,osd_pg_creating=1703736855,osd_snap=1115056769}) 2024-09-18T11:52:04.159 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.160 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=1} crc {logm=3076416300,mdsmap=1531151339,mgr=3797927047,mgr_metadata=454364145,mgrstat=1264896640,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=1703736855,osd_snap=3671887149,osdmap=1214209872}) 2024-09-18T11:52:04.160 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.154+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=2} crc {logm=930132462,mdsmap=1531151339,mgr=3332224566,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1264896640,mon_config_key=3035149395,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=1703736855,osd_snap=3687295090,osdmap=2030908468}) 2024-09-18T11:52:04.161 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.161 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2196831193}) 2024-09-18T11:52:04.161 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=996556322}) 2024-09-18T11:52:04.161 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.161 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2196831193}) 2024-09-18T11:52:04.161 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=500473681}) 2024-09-18T11:52:04.163 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.163 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3534627686}) 2024-09-18T11:52:04.163 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2395330102}) 2024-09-18T11:52:04.163 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.163 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3534627686}) 2024-09-18T11:52:04.164 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.158+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=90536586}) 2024-09-18T11:52:04.164 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.165 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2256002563}) 2024-09-18T11:52:04.165 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3516707248}) 2024-09-18T11:52:04.165 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.165 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2256002563}) 2024-09-18T11:52:04.165 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1850075041}) 2024-09-18T11:52:04.167 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.167 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=941132149}) 2024-09-18T11:52:04.167 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1207981361}) 2024-09-18T11:52:04.167 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.167 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=941132149}) 2024-09-18T11:52:04.167 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.162+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4051897094}) 2024-09-18T11:52:04.175 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.170+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.175 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.170+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=94} crc {osdmap=1943985526}) 2024-09-18T11:52:04.175 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.170+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2880466648}) 2024-09-18T11:52:04.175 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.170+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:52:04.175 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.170+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=94} crc {osdmap=1943985526}) 2024-09-18T11:52:04.175 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:52:04.170+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3793810015}) 2024-09-18T11:52:05.112 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:52:05.113 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.218+0000 7f3b2d4e4700 10 monclient: get_monmap_and_config 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.218+0000 7f3b2d4e4700 10 monclient: build_initial_monmap 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.218+0000 7f3b2d4e4700 10 monclient: monmap: 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:05.224266+0000 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:05.224266+0000 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.224 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:05.225 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:05.225 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:05.225 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 1 Processor -- start 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 1 -- start start 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 10 monclient: init 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 10 monclient(hunting): picked mon.noname-c con 0x7f3b280a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 10 monclient(hunting): picked mon.noname-b con 0x7f3b2806c280 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 10 monclient(hunting): picked mon.noname-a con 0x7f3b2806c670 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2806e950 con 0x7f3b2806c670 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b280a2270 con 0x7f3b280a1ea0 2024-09-18T11:52:05.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2806d130 con 0x7f3b2806c280 2024-09-18T11:52:05.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:05.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2d4e4700 10 monclient(hunting): authenticate will time out at 1297.065141s 2024-09-18T11:52:05.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b277fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3b2806c280 0x7f3b28135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44640/0 (socket says 172.21.15.112:44640) 2024-09-18T11:52:05.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b277fe700 1 -- 172.21.15.112:0/1668413538 learned_addr learned my addr 172.21.15.112:0/1668413538 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b277fe700 1 -- 172.21.15.112:0/1668413538 >> v1:172.21.15.112:6790/0 conn(0x7f3b2806c280 legacy=0x7f3b28135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b2cce3700 1 -- 172.21.15.112:0/1668413538 >> v1:172.21.15.112:6789/0 conn(0x7f3b2806c670 legacy=0x7f3b281388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332446481 0 0) 0x7f3b2806d130 con 0x7f3b2806c280 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b26ffd700 10 monclient(hunting): my global_id is 104135 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b10003570 con 0x7f3b2806c280 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.222+0000 7f3b27fff700 1 -- 172.21.15.112:0/1668413538 >> v1:172.21.15.129:6789/0 conn(0x7f3b280a1ea0 legacy=0x7f3b2806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854389842 0 0) 0x7f3b2806e950 con 0x7f3b2806c670 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient(hunting): my global_id is 94257 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2806d130 con 0x7f3b2806c670 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440206240 0 0) 0x7f3b10003570 con 0x7f3b2806c280 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 >> v1:172.21.15.129:6789/0 conn(0x7f3b280a1ea0 legacy=0x7f3b2806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 >> v1:172.21.15.112:6789/0 conn(0x7f3b2806c670 legacy=0x7f3b281388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 monclient: found mon.noname-b 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2809e370 con 0x7f3b2806c280 2024-09-18T11:52:05.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: _finish_auth 0 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3b18001a00 con 0x7f3b2806c280 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: dump: 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.231 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 5 monclient: authenticate success, global_id 104135 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b18001be0 con 0x7f3b2806c280 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 4 set_mon_vals no callback set 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 1 -- 172.21.15.112:0/1668413538 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3b18004a90 con 0x7f3b2806c280 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b26ffd700 10 monclient: dump: 2024-09-18T11:52:05.232 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.233 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 10 monclient: shutdown 2024-09-18T11:52:05.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:05.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/1668413538 >> v1:172.21.15.112:6790/0 conn(0x7f3b2806c280 legacy=0x7f3b28135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/1668413538 shutdown_connections 2024-09-18T11:52:05.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/1668413538 >> 172.21.15.112:0/1668413538 conn(0x7f3b28067da0 msgr2=0x7f3b280a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:05.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/1668413538 shutdown_connections 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/1668413538 wait complete. 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 10 monclient: build_initial_monmap 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.226+0000 7f3b2d4e4700 10 monclient: monmap: 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:05.241 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 librados: starting msgr at 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 librados: starting objecter 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 Processor -- start 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 -- start start 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 librados: setting wanted keys 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 librados: calling monclient init 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 10 monclient: init 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 10 monclient(hunting): picked mon.noname-a con 0x7f3b2806c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 10 monclient(hunting): picked mon.noname-c con 0x7f3b2806c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:05.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 10 monclient(hunting): picked mon.noname-b con 0x7f3b280a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2806e1a0 con 0x7f3b2806c280 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b28132ea0 con 0x7f3b280a1ea0 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b28119600 con 0x7f3b2806c670 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b2d4e4700 10 monclient(hunting): authenticate will time out at 1297.073680s 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b277fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3b2806c670 0x7f3b28132260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44642/0 (socket says 172.21.15.112:44642) 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.230+0000 7f3b277fe700 1 -- 172.21.15.112:0/862028678 learned_addr learned my addr 172.21.15.112:0/862028678 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b27fff700 1 -- 172.21.15.112:0/862028678 >> v1:172.21.15.112:6789/0 conn(0x7f3b2806c280 legacy=0x7f3b2813f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 client.?.objecter ms_handle_connect 0x7f3b2806c280 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 client.?.objecter ms_handle_connect 0x7f3b2806c670 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b277fe700 1 -- 172.21.15.112:0/862028678 >> v1:172.21.15.112:6790/0 conn(0x7f3b2806c670 legacy=0x7f3b28132260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2cce3700 1 -- 172.21.15.112:0/862028678 >> v1:172.21.15.129:6789/0 conn(0x7f3b280a1ea0 legacy=0x7f3b28147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 client.?.objecter ms_handle_connect 0x7f3b280a1ea0 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141084530 0 0) 0x7f3b2806e1a0 con 0x7f3b2806c280 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): my global_id is 94260 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b14004f10 con 0x7f3b2806c280 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355671621 0 0) 0x7f3b28119600 con 0x7f3b2806c670 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): my global_id is 104138 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2806e1a0 con 0x7f3b2806c670 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843395943 0 0) 0x7f3b28132ea0 con 0x7f3b280a1ea0 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): my global_id is 18178 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b28119600 con 0x7f3b280a1ea0 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1573934366 0 0) 0x7f3b14004f10 con 0x7f3b2806c280 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b28132ea0 con 0x7f3b2806c280 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3206031318 0 0) 0x7f3b2806e1a0 con 0x7f3b2806c670 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b14004f10 con 0x7f3b2806c670 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3b1c001c80 con 0x7f3b2806c280 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): dump: 2024-09-18T11:52:05.245 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3b180049d0 con 0x7f3b2806c670 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient(hunting): dump: 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2405584293 0 0) 0x7f3b14004f10 con 0x7f3b2806c670 2024-09-18T11:52:05.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 >> v1:172.21.15.129:6789/0 conn(0x7f3b280a1ea0 legacy=0x7f3b28147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 >> v1:172.21.15.112:6789/0 conn(0x7f3b2806c280 legacy=0x7f3b2813f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient: _finish_hunting 0 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 monclient: found mon.c 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28132970 con 0x7f3b2806c670 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient: _finish_auth 0 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 5 monclient: authenticate success, global_id 104138 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 10 monclient: _renew_subs 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b28139b70 con 0x7f3b2806c670 2024-09-18T11:52:05.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 10 client.104138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 10 monclient: _renew_subs 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b281193d0 con 0x7f3b2806c670 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 1 librados: init done 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b18004b30 con 0x7f3b2806c670 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3b180051e0 con 0x7f3b2806c670 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b267fc700 4 set_mon_vals no callback set 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 monclient: dump: 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.249 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f3b1800fa60 con 0x7f3b2806c670 2024-09-18T11:52:05.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b080050b0 con 0x7f3b2806c670 2024-09-18T11:52:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(342..342 src has 1..342) v4 ==== 5393+0+0 (unknown 1437224965 0 0) 0x7f3b18043fa0 con 0x7f3b2806c670 2024-09-18T11:52:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 10 client.104138.objecter ms_dispatch 0x7f3b28000fd0 osd_map(342..342 src has 1..342) v4 2024-09-18T11:52:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 3 client.104138.objecter handle_osd_map got epochs [342,342] > 0 2024-09-18T11:52:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 3 client.104138.objecter handle_osd_map decoding full epoch 342 2024-09-18T11:52:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b0ffff700 20 client.104138.objecter dump_active .. 0 homeless 2024-09-18T11:52:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.234+0000 7f3b27fff700 10 monclient: get_auth_request con 0x7f3b14038510 auth_method 0 2024-09-18T11:52:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.238+0000 7f3b0ffff700 10 client.104138.objecter ms_handle_connect 0x7f3b14038510 2024-09-18T11:52:05.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.242+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f3b18014bd0 con 0x7f3b2806c670 2024-09-18T11:52:05.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.242+0000 7f3b0ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.242+0000 7f3b0ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:05.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.466+0000 7f3b2d4e4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:52:05.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.466+0000 7f3b2d4e4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:52:05.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.466+0000 7f3b2d4e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.466+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3b080058e0 con 0x7f3b2806c670 2024-09-18T11:52:05.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.466+0000 7f3b0ffff700 1 -- 172.21.15.112:0/862028678 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 895057067) 0x7f3b180145d0 con 0x7f3b2806c670 2024-09-18T11:52:05.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.466+0000 7f3b0ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:52:05.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.466+0000 7f3b0ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:05.471 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:52:05.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 10 librados: watch_flush enter 2024-09-18T11:52:05.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 10 librados: watch_flush exit 2024-09-18T11:52:05.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 20 client.104138.objecter shutdown clearing up homeless session... 2024-09-18T11:52:05.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 10 client.104138.objecter successfully canceled tick 2024-09-18T11:52:05.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 >> v1:172.21.15.129:6812/20806 conn(0x7f3b14038510 legacy=0x7f3b1403a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 10 monclient: shutdown 2024-09-18T11:52:05.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:05.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 >> v1:172.21.15.112:6790/0 conn(0x7f3b2806c670 legacy=0x7f3b28132260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 shutdown_connections 2024-09-18T11:52:05.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 >> 172.21.15.112:0/862028678 conn(0x7f3b28067da0 msgr2=0x7f3b2809fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:05.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 shutdown_connections 2024-09-18T11:52:05.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 1 -- 172.21.15.112:0/862028678 wait complete. 2024-09-18T11:52:05.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.470+0000 7f3b2d4e4700 1 librados: shutdown 2024-09-18T11:52:05.484 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:52:05.484 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T11:52:05.484 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:52:05.485 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient: get_monmap_and_config 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient: build_initial_monmap 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient: monmap: 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:05.589340+0000 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:05.589340+0000 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:05.590 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:05.590 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:05.590 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 1 Processor -- start 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 1 -- start start 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient: init 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient(hunting): picked mon.noname-c con 0x7f55900a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient(hunting): picked mon.noname-a con 0x7f5590096a60 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient(hunting): picked mon.noname-b con 0x7f559013ca20 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559006e910 con 0x7f5590096a60 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559006e630 con 0x7f55900a1ea0 2024-09-18T11:52:05.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55900a2270 con 0x7f559013ca20 2024-09-18T11:52:05.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:05.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f5594f1a700 10 monclient(hunting): authenticate will time out at 1297.430180s 2024-09-18T11:52:05.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f558ffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f559013ca20 0x7f559013ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44644/0 (socket says 172.21.15.112:44644) 2024-09-18T11:52:05.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.586+0000 7f558ffff700 1 -- 172.21.15.112:0/3659192394 learned_addr learned my addr 172.21.15.112:0/3659192394 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:05.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558effd700 1 -- 172.21.15.112:0/3659192394 >> v1:172.21.15.112:6789/0 conn(0x7f5590096a60 legacy=0x7f559006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558ffff700 1 -- 172.21.15.112:0/3659192394 >> v1:172.21.15.112:6790/0 conn(0x7f559013ca20 legacy=0x7f559013ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558f7fe700 1 -- 172.21.15.112:0/3659192394 >> v1:172.21.15.129:6789/0 conn(0x7f55900a1ea0 legacy=0x7f559006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102481832 0 0) 0x7f55900a2270 con 0x7f559013ca20 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): my global_id is 104141 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5578003570 con 0x7f559013ca20 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886906199 0 0) 0x7f559006e910 con 0x7f5590096a60 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): my global_id is 94263 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55900a2270 con 0x7f5590096a60 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601611339 0 0) 0x7f559006e630 con 0x7f55900a1ea0 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient(hunting): my global_id is 18181 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559006e910 con 0x7f55900a1ea0 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1144336006 0 0) 0x7f55900a2270 con 0x7f5590096a60 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 >> v1:172.21.15.112:6790/0 conn(0x7f559013ca20 legacy=0x7f559013ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 >> v1:172.21.15.129:6789/0 conn(0x7f55900a1ea0 legacy=0x7f559006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 monclient: found mon.noname-a 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559006eed0 con 0x7f5590096a60 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 5 monclient: authenticate success, global_id 94263 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5588001c70 con 0x7f5590096a60 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: dump: 2024-09-18T11:52:05.597 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:05.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5588001f00 con 0x7f5590096a60 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 1 -- 172.21.15.112:0/3659192394 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5588002420 con 0x7f5590096a60 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 4 set_mon_vals no callback set 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f558e7fc700 10 monclient: dump: 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.599 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 10 monclient: shutdown 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 1 -- 172.21.15.112:0/3659192394 >> v1:172.21.15.112:6789/0 conn(0x7f5590096a60 legacy=0x7f559006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 1 -- 172.21.15.112:0/3659192394 shutdown_connections 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 1 -- 172.21.15.112:0/3659192394 >> 172.21.15.112:0/3659192394 conn(0x7f5590067da0 msgr2=0x7f55900a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 1 -- 172.21.15.112:0/3659192394 shutdown_connections 2024-09-18T11:52:05.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 1 -- 172.21.15.112:0/3659192394 wait complete. 2024-09-18T11:52:05.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 10 monclient: build_initial_monmap 2024-09-18T11:52:05.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.590+0000 7f5594f1a700 10 monclient: monmap: 2024-09-18T11:52:05.606 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 librados: starting msgr at 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 librados: starting objecter 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 Processor -- start 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 -- start start 2024-09-18T11:52:05.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 librados: setting wanted keys 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 librados: calling monclient init 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 10 monclient: init 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 10 monclient(hunting): picked mon.noname-b con 0x7f5590096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 10 monclient(hunting): picked mon.noname-c con 0x7f55900a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 10 monclient(hunting): picked mon.noname-a con 0x7f559013ca20 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559006e110 con 0x7f559013ca20 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559009dca0 con 0x7f5590096a60 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55901268e0 con 0x7f55900a1ea0 2024-09-18T11:52:05.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f5594f1a700 10 monclient(hunting): authenticate will time out at 1297.437899s 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f558effd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f55900a1ea0 0x7f55901467a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44652/0 (socket says 172.21.15.112:44652) 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.594+0000 7f558effd700 1 -- 172.21.15.112:0/3763035211 learned_addr learned my addr 172.21.15.112:0/3763035211 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f558effd700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6790/0 conn(0x7f55900a1ea0 legacy=0x7f55901467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 client.?.objecter ms_handle_connect 0x7f55900a1ea0 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f558ffff700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6789/0 conn(0x7f559013ca20 legacy=0x7f559014e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 client.?.objecter ms_handle_connect 0x7f559013ca20 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f558effd700 1 --1- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6790/0 conn(0x7f55900a1ea0 0x7f55901467a0 :-1 s=OPENED pgs=8959 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f55901268e0 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f558f7fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.129:6789/0 conn(0x7f5590096a60 legacy=0x7f5590146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 client.?.objecter ms_handle_connect 0x7f5590096a60 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497561153 0 0) 0x7f559006e110 con 0x7f559013ca20 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): my global_id is 94266 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c004f10 con 0x7f559013ca20 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839365292 0 0) 0x7f559009dca0 con 0x7f5590096a60 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): my global_id is 18184 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559006e110 con 0x7f5590096a60 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3798253989 0 0) 0x7f557c004f10 con 0x7f559013ca20 2024-09-18T11:52:05.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f559009dca0 con 0x7f559013ca20 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5580004830 con 0x7f559013ca20 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): dump: 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.611 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3186873974 0 0) 0x7f559009dca0 con 0x7f559013ca20 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6790/0 conn(0x7f55900a1ea0 legacy=0x7f55901467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.129:6789/0 conn(0x7f5590096a60 legacy=0x7f5590146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient: _finish_hunting 0 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 monclient: found mon.a 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5590125330 con 0x7f559013ca20 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient: _finish_auth 0 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 5 monclient: authenticate success, global_id 94266 2024-09-18T11:52:05.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400380585 0 0) 0x7f55901268e0 con 0x7f55900a1ea0 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 10 monclient: _renew_subs 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5590126ea0 con 0x7f559013ca20 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 10 client.94266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 10 monclient: _renew_subs 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5590127d30 con 0x7f559013ca20 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 1 librados: init done 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5580004c40 con 0x7f559013ca20 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5580004f50 con 0x7f559013ca20 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f558dffb700 4 set_mon_vals no callback set 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 monclient: dump: 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5570006590 con 0x7f5570002840 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f558000f7d0 con 0x7f559013ca20 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(342..342 src has 1..342) v4 ==== 5393+0+0 (unknown 1437224965 0 0) 0x7f5580043f20 con 0x7f559013ca20 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 10 client.94266.objecter ms_dispatch 0x7f5590000fd0 osd_map(342..342 src has 1..342) v4 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 3 client.94266.objecter handle_osd_map got epochs [342,342] > 0 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 3 client.94266.objecter handle_osd_map decoding full epoch 342 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f55777fe700 20 client.94266.objecter dump_active .. 0 homeless 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.598+0000 7f558effd700 10 monclient: get_auth_request con 0x7f557c037e70 auth_method 0 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f558f7fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6789/0 conn(0x7f5570002840 legacy=0x7f5570004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 client.94266.objecter ms_handle_connect 0x7f5570002840 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 client.94266.objecter resend_mon_ops 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099560079 0 0) 0x7f5570006590 con 0x7f5570002840 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 monclient(hunting): my global_id is 94269 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55901268e0 con 0x7f5570002840 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4204191375 0 0) 0x7f55901268e0 con 0x7f5570002840 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5570006590 con 0x7f5570002840 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5584002310 con 0x7f5570002840 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 monclient: dump: 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.616 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.617 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.617 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.617 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.617 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1697908973 0 0) 0x7f5570006590 con 0x7f5570002840 2024-09-18T11:52:05.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f557000a140 con 0x7f5570002840 2024-09-18T11:52:05.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 client.94266.objecter ms_handle_connect 0x7f557c037e70 2024-09-18T11:52:05.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5584005810 con 0x7f5570002840 2024-09-18T11:52:05.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:05.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6789/0 conn(0x7f5570002840 legacy=0x7f5570004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.602+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6789/0 conn(0x7f5570002840 legacy=0x7f5570004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:05.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f5594f1a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:52:05.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5570009f10 con 0x7f5570006900 2024-09-18T11:52:05.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f558f7fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6789/0 conn(0x7f5570006900 legacy=0x7f5570006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f55777fe700 10 client.94266.objecter ms_handle_connect 0x7f5570006900 2024-09-18T11:52:05.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f55777fe700 10 client.94266.objecter resend_mon_ops 2024-09-18T11:52:05.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450275506 0 0) 0x7f5570009f10 con 0x7f5570006900 2024-09-18T11:52:05.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f55777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f55777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f55777fe700 10 monclient(hunting): my global_id is 94272 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.614+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5570006590 con 0x7f5570006900 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3406719481 0 0) 0x7f5570006590 con 0x7f5570006900 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5570009f10 con 0x7f5570006900 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f55840028a0 con 0x7f5570006900 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 10 monclient: dump: 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.621 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3157497815 0 0) 0x7f5570009f10 con 0x7f5570006900 2024-09-18T11:52:05.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5570008760 con 0x7f5570006900 2024-09-18T11:52:05.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4003 (unknown 0 0 2314916841) 0x7f5584002c50 con 0x7f5570006900 2024-09-18T11:52:05.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:52:05.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:05.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6789/0 conn(0x7f5570006900 legacy=0x7f5570006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f55777fe700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6789/0 conn(0x7f5570006900 legacy=0x7f5570006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:05.624 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:52:05.624 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:05.624 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:05.624 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:52:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 36, 2024-09-18T11:52:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:52:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:52:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:52:05.625 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:52:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:52:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:52:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:52:05.626 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.627 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:52:05.628 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:52:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.629 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.630 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:52:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:52:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:52:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:52:05.631 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:52:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:52:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:52:05.632 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:52:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:52:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:52:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:52:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:52:05.633 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:52:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:52:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.634 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.635 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.635 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.635 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.635 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.635 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.636 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.636 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:52:05.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.636 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:52:05.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.636 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:05.637 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:05.637 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.637 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.637 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.638 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:52:05.638 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.638 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.638 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.639 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:52:05.639 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:52:05.639 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.639 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.639 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.640 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:05.640 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:05.640 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.640 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.641 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.641 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:52:05.641 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.641 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.642 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:52:05.642 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:52:05.642 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.642 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.642 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:52:05.643 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:52:05.644 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:52:05.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:52:05.646 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 10 librados: watch_flush enter 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 10 librados: watch_flush exit 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 20 client.94266.objecter shutdown clearing up homeless session... 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 10 client.94266.objecter successfully canceled tick 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.129:6812/20806 conn(0x7f557c037e70 legacy=0x7f557c03a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 10 monclient: shutdown 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 >> v1:172.21.15.112:6789/0 conn(0x7f559013ca20 legacy=0x7f559014e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f558ffff700 1 -- 172.21.15.112:0/3763035211 reap_dead start 2024-09-18T11:52:05.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f558ffff700 1 -- 172.21.15.112:0/3763035211 reap_dead start 2024-09-18T11:52:05.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 shutdown_connections 2024-09-18T11:52:05.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 >> 172.21.15.112:0/3763035211 conn(0x7f5590067da0 msgr2=0x7f559009f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:05.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 shutdown_connections 2024-09-18T11:52:05.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 1 -- 172.21.15.112:0/3763035211 wait complete. 2024-09-18T11:52:05.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.618+0000 7f5594f1a700 1 librados: shutdown 2024-09-18T11:52:05.649 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:52:05.649 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:52:05.649 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:52:05.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient: get_monmap_and_config 2024-09-18T11:52:05.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient: build_initial_monmap 2024-09-18T11:52:05.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient: monmap: 2024-09-18T11:52:05.766 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:05.766 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.766 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:05.766106+0000 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:05.766106+0000 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 1 Processor -- start 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 1 -- start start 2024-09-18T11:52:05.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient: init 2024-09-18T11:52:05.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:05.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:52:05.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient(hunting): picked mon.noname-a con 0x7fa76c0a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:05.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient(hunting): picked mon.noname-b con 0x7fa76c06c280 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient(hunting): picked mon.noname-c con 0x7fa76c06c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c06e950 con 0x7fa76c0a1ea0 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c0a2270 con 0x7fa76c06c670 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c06d130 con 0x7fa76c06c280 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa771a86700 10 monclient(hunting): authenticate will time out at 1297.606926s 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa763fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c280 0x7fa76c135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44664/0 (socket says 172.21.15.112:44664) 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa763fff700 1 -- 172.21.15.112:0/4227496175 learned_addr learned my addr 172.21.15.112:0/4227496175 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:05.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.762+0000 7fa770a84700 1 --1- 172.21.15.112:0/4227496175 >> v1:172.21.15.112:6789/0 conn(0x7fa76c0a1ea0 0x7fa76c06bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:45520/0 (socket says 172.21.15.112:45520) 2024-09-18T11:52:05.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa763fff700 1 -- 172.21.15.112:0/4227496175 >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c280 legacy=0x7fa76c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa770a84700 1 -- 172.21.15.112:0/4227496175 >> v1:172.21.15.112:6789/0 conn(0x7fa76c0a1ea0 legacy=0x7fa76c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771285700 1 -- 172.21.15.112:0/4227496175 >> v1:172.21.15.129:6789/0 conn(0x7fa76c06c670 legacy=0x7fa76c1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588806932 0 0) 0x7fa76c06d130 con 0x7fa76c06c280 2024-09-18T11:52:05.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): my global_id is 104147 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa754003570 con 0x7fa76c06c280 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805211139 0 0) 0x7fa76c06e950 con 0x7fa76c0a1ea0 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): my global_id is 94275 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c06d130 con 0x7fa76c0a1ea0 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469235208 0 0) 0x7fa76c0a2270 con 0x7fa76c06c670 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient(hunting): my global_id is 18187 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c06e950 con 0x7fa76c06c670 2024-09-18T11:52:05.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1686422983 0 0) 0x7fa754003570 con 0x7fa76c06c280 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 >> v1:172.21.15.129:6789/0 conn(0x7fa76c06c670 legacy=0x7fa76c1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 >> v1:172.21.15.112:6789/0 conn(0x7fa76c0a1ea0 legacy=0x7fa76c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: _finish_hunting 0 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 monclient: found mon.noname-b 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c09e370 con 0x7fa76c06c280 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: _finish_auth 0 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 5 monclient: authenticate success, global_id 104147 2024-09-18T11:52:05.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa75c001a00 con 0x7fa76c06c280 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: dump: 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.774 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa75c001b60 con 0x7fa76c06c280 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 4 set_mon_vals no callback set 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 1 -- 172.21.15.112:0/4227496175 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa75c0048f0 con 0x7fa76c06c280 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa7637fe700 10 monclient: dump: 2024-09-18T11:52:05.775 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 10 monclient: shutdown 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 1 -- 172.21.15.112:0/4227496175 >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c280 legacy=0x7fa76c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 1 -- 172.21.15.112:0/4227496175 shutdown_connections 2024-09-18T11:52:05.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 1 -- 172.21.15.112:0/4227496175 >> 172.21.15.112:0/4227496175 conn(0x7fa76c067da0 msgr2=0x7fa76c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:05.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 1 -- 172.21.15.112:0/4227496175 shutdown_connections 2024-09-18T11:52:05.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.766+0000 7fa771a86700 1 -- 172.21.15.112:0/4227496175 wait complete. 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient: build_initial_monmap 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient: monmap: 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:05.783 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 librados: starting msgr at 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 librados: starting objecter 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 Processor -- start 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 -- start start 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 librados: setting wanted keys 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 librados: calling monclient init 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient: init 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient(hunting): picked mon.noname-a con 0x7fa76c06c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:05.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient(hunting): picked mon.noname-c con 0x7fa76c06c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient(hunting): picked mon.noname-b con 0x7fa76c0a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c132e00 con 0x7fa76c06c280 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c11f2b0 con 0x7fa76c0a1ea0 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c140120 con 0x7fa76c06c670 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa771a86700 10 monclient(hunting): authenticate will time out at 1297.614215s 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa763fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c670 0x7fa76c144520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44666/0 (socket says 172.21.15.112:44666) 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.770+0000 7fa763fff700 1 -- 172.21.15.112:0/2362035130 learned_addr learned my addr 172.21.15.112:0/2362035130 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa763fff700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c670 legacy=0x7fa76c144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 client.?.objecter ms_handle_connect 0x7fa76c06c670 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 client.?.objecter ms_handle_connect 0x7fa76c06c280 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa770a84700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6789/0 conn(0x7fa76c06c280 legacy=0x7fa76c131e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa763fff700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c670 legacy=0x7fa76c144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa763fff700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c670 legacy=0x7fa76c144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa763fff700 1 --1- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c670 0x7fa76c144520 :-1 s=OPENED pgs=8961 cs=1 l=1).handle_message read tag failed 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa763fff700 1 --1- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c670 0x7fa76c144520 :-1 s=OPENED pgs=8961 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6790/0 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901825147 0 0) 0x7fa76c132e00 con 0x7fa76c06c280 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): my global_id is 94278 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa758004f10 con 0x7fa76c06c280 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771285700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.129:6789/0 conn(0x7fa76c0a1ea0 legacy=0x7fa76c147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771285700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.129:6789/0 conn(0x7fa76c0a1ea0 legacy=0x7fa76c147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T11:52:05.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 client.?.objecter ms_handle_connect 0x7fa76c0a1ea0 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771285700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.129:6789/0 conn(0x7fa76c0a1ea0 legacy=0x7fa76c147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771285700 1 --1- 172.21.15.112:0/2362035130 >> v1:172.21.15.129:6789/0 conn(0x7fa76c0a1ea0 0x7fa76c147c50 :-1 s=OPENED pgs=8994 cs=1 l=1).handle_message read tag failed 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771285700 1 --1- 172.21.15.112:0/2362035130 >> v1:172.21.15.129:6789/0 conn(0x7fa76c0a1ea0 0x7fa76c147c50 :-1 s=OPENED pgs=8994 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.129:6789/0 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2810356468 0 0) 0x7fa758004f10 con 0x7fa76c06c280 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa76c132e00 con 0x7fa76c06c280 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa768001c50 con 0x7fa76c06c280 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): dump: 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.787 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3419300856 0 0) 0x7fa76c132e00 con 0x7fa76c06c280 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa76c06c670 legacy=0x7fa76c144520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.129:6789/0 conn(0x7fa76c0a1ea0 legacy=0x7fa76c147c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:52:05.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 monclient: found mon.a 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c11ef40 con 0x7fa76c06c280 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient: _finish_auth 0 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 5 monclient: authenticate success, global_id 94278 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 10 monclient: _renew_subs 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76c11d480 con 0x7fa76c06c280 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 10 client.94278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 10 monclient: _renew_subs 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:05.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76c132530 con 0x7fa76c06c280 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 1 librados: init done 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa768004be0 con 0x7fa76c06c280 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa768004f30 con 0x7fa76c06c280 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa762ffd700 4 set_mon_vals no callback set 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 monclient: dump: 2024-09-18T11:52:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa74c006590 con 0x7fa74c002840 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa768005090 con 0x7fa76c06c280 2024-09-18T11:52:05.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(342..342 src has 1..342) v4 ==== 5393+0+0 (unknown 1437224965 0 0) 0x7fa768043ee0 con 0x7fa76c06c280 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 10 client.94278.objecter ms_dispatch 0x7fa76c000fd0 osd_map(342..342 src has 1..342) v4 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 3 client.94278.objecter handle_osd_map got epochs [342,342] > 0 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 3 client.94278.objecter handle_osd_map decoding full epoch 342 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa760ff9700 20 client.94278.objecter dump_active .. 0 homeless 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.774+0000 7fa771285700 10 monclient: get_auth_request con 0x7fa758037f80 auth_method 0 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 client.94278.objecter ms_handle_connect 0x7fa74c002840 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa763fff700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa74c002840 legacy=0x7fa74c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 client.94278.objecter resend_mon_ops 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029410549 0 0) 0x7fa74c006590 con 0x7fa74c002840 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 monclient(hunting): my global_id is 104150 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c132e00 con 0x7fa74c002840 2024-09-18T11:52:05.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2973835544 0 0) 0x7fa76c132e00 con 0x7fa74c002840 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa74c006590 con 0x7fa74c002840 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa75c0015e0 con 0x7fa74c002840 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 monclient: dump: 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.793 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3294027662 0 0) 0x7fa74c006590 con 0x7fa74c002840 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa74c00a140 con 0x7fa74c002840 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 client.94278.objecter ms_handle_connect 0x7fa758037f80 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fa75c006fc0 con 0x7fa74c002840 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa74c002840 legacy=0x7fa74c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.778+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa74c002840 legacy=0x7fa74c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:05.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.786+0000 7fa771a86700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:52:05.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.786+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa74c009670 con 0x7fa74c0090e0 2024-09-18T11:52:05.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa763fff700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa74c0090e0 legacy=0x7fa74c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 10 client.94278.objecter ms_handle_connect 0x7fa74c0090e0 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 10 client.94278.objecter resend_mon_ops 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362981871 0 0) 0x7fa74c009670 con 0x7fa74c0090e0 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 10 monclient(hunting): my global_id is 104153 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa74c006590 con 0x7fa74c0090e0 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2136549199 0 0) 0x7fa74c006590 con 0x7fa74c0090e0 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa74c009670 con 0x7fa74c0090e0 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa75c005390 con 0x7fa74c0090e0 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:05.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 10 monclient: dump: 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3522754940 0 0) 0x7fa74c009670 con 0x7fa74c0090e0 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.790+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa74c009930 con 0x7fa74c0090e0 2024-09-18T11:52:05.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 246611326) 0x7fa75c0071d0 con 0x7fa74c0090e0 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa760ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa760ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa74c0090e0 legacy=0x7fa74c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa760ff9700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6790/0 conn(0x7fa74c0090e0 legacy=0x7fa74c008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:52:05.799 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 36, 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.800 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:52:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:52:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:52:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:52:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:52:05.811 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 10 librados: watch_flush enter 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 10 librados: watch_flush exit 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 20 client.94278.objecter shutdown clearing up homeless session... 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 10 client.94278.objecter successfully canceled tick 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.129:6812/20806 conn(0x7fa758037f80 legacy=0x7fa75803a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 10 monclient: shutdown 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 >> v1:172.21.15.112:6789/0 conn(0x7fa76c06c280 legacy=0x7fa76c131e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771285700 1 -- 172.21.15.112:0/2362035130 reap_dead start 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771285700 1 -- 172.21.15.112:0/2362035130 reap_dead start 2024-09-18T11:52:05.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 shutdown_connections 2024-09-18T11:52:05.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 >> 172.21.15.112:0/2362035130 conn(0x7fa76c067da0 msgr2=0x7fa76c133010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:05.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 shutdown_connections 2024-09-18T11:52:05.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 1 -- 172.21.15.112:0/2362035130 wait complete. 2024-09-18T11:52:05.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.794+0000 7fa771a86700 1 librados: shutdown 2024-09-18T11:52:05.816 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:52:05.816 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:52:05.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient: get_monmap_and_config 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient: build_initial_monmap 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient: monmap: 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:05.930114+0000 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:05.930114+0000 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:05.930 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:05.931 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:05.931 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 1 Processor -- start 2024-09-18T11:52:05.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 1 -- start start 2024-09-18T11:52:05.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient: init 2024-09-18T11:52:05.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:05.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:52:05.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient(hunting): picked mon.noname-a con 0x7fa8a00a21c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient(hunting): picked mon.noname-c con 0x7fa8a009fdf0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient(hunting): picked mon.noname-b con 0x7fa8a00a01f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a006b7c0 con 0x7fa8a00a21c0 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a00a2590 con 0x7fa8a009fdf0 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a009beb0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa8a5062700 10 monclient(hunting): authenticate will time out at 1297.771215s 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa89f59e700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa8a00a01f0 0x7fa8a013f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44692/0 (socket says 172.21.15.112:44692) 2024-09-18T11:52:05.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.926+0000 7fa89f59e700 1 -- 172.21.15.112:0/3919032134 learned_addr learned my addr 172.21.15.112:0/3919032134 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:05.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89f59e700 1 -- 172.21.15.112:0/3919032134 >> v1:172.21.15.112:6790/0 conn(0x7fa8a00a01f0 legacy=0x7fa8a013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89ed9d700 1 -- 172.21.15.112:0/3919032134 >> v1:172.21.15.112:6789/0 conn(0x7fa8a00a21c0 legacy=0x7fa8a009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89659c700 1 -- 172.21.15.112:0/3919032134 >> v1:172.21.15.129:6789/0 conn(0x7fa8a009fdf0 legacy=0x7fa8a013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840796198 0 0) 0x7fa8a009beb0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): my global_id is 104156 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa880003570 con 0x7fa8a00a01f0 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593742890 0 0) 0x7fa8a006b7c0 con 0x7fa8a00a21c0 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): my global_id is 94281 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a009beb0 con 0x7fa8a00a21c0 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832398746 0 0) 0x7fa8a00a2590 con 0x7fa8a009fdf0 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient(hunting): my global_id is 18190 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a006b7c0 con 0x7fa8a009fdf0 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1023860467 0 0) 0x7fa880003570 con 0x7fa8a00a01f0 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 >> v1:172.21.15.129:6789/0 conn(0x7fa8a009fdf0 legacy=0x7fa8a013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 >> v1:172.21.15.112:6789/0 conn(0x7fa8a00a21c0 legacy=0x7fa8a009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: _finish_hunting 0 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 monclient: found mon.noname-b 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a00a0fb0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: _finish_auth 0 2024-09-18T11:52:05.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa898001cc0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa8a5062700 5 monclient: authenticate success, global_id 104156 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa8a5062700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: dump: 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.938 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa8a5062700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa898001f50 con 0x7fa8a00a01f0 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 1 -- 172.21.15.112:0/3919032134 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa898004e10 con 0x7fa8a00a01f0 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa89e59c700 10 monclient: dump: 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.939 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa8a5062700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:05.940 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.930+0000 7fa8a5062700 4 set_mon_vals no callback set 2024-09-18T11:52:05.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient: shutdown 2024-09-18T11:52:05.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:05.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- 172.21.15.112:0/3919032134 >> v1:172.21.15.112:6790/0 conn(0x7fa8a00a01f0 legacy=0x7fa8a013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- 172.21.15.112:0/3919032134 shutdown_connections 2024-09-18T11:52:05.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- 172.21.15.112:0/3919032134 >> 172.21.15.112:0/3919032134 conn(0x7fa8a00680d0 msgr2=0x7fa8a00a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:05.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- 172.21.15.112:0/3919032134 shutdown_connections 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- 172.21.15.112:0/3919032134 wait complete. 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient: build_initial_monmap 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient: monmap: 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 librados: starting msgr at 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 librados: starting objecter 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 Processor -- start 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- start start 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 librados: setting wanted keys 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 librados: calling monclient init 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient: init 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient(hunting): picked mon.noname-a con 0x7fa8a009fdf0 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient(hunting): picked mon.noname-c con 0x7fa8a00a01f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient(hunting): picked mon.noname-b con 0x7fa8a00a21c0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a0126fb0 con 0x7fa8a009fdf0 2024-09-18T11:52:05.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a0142340 con 0x7fa8a00a21c0 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a0143070 con 0x7fa8a00a01f0 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa89659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa8a00a01f0 0x7fa8a014ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:44706/0 (socket says 172.21.15.112:44706) 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa89659c700 1 -- 172.21.15.112:0/1181152561 learned_addr learned my addr 172.21.15.112:0/1181152561 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.934+0000 7fa8a5062700 10 monclient(hunting): authenticate will time out at 1297.779826s 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa89659c700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.112:6790/0 conn(0x7fa8a00a01f0 legacy=0x7fa8a014ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 client.?.objecter ms_handle_connect 0x7fa8a00a01f0 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa89ed9d700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.112:6789/0 conn(0x7fa8a009fdf0 legacy=0x7fa8a0141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 client.?.objecter ms_handle_connect 0x7fa8a009fdf0 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488303079 0 0) 0x7fa8a0143070 con 0x7fa8a00a01f0 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): my global_id is 104159 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa89f59e700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6789/0 conn(0x7fa8a00a21c0 legacy=0x7fa8a014e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884004f10 con 0x7fa8a00a01f0 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 client.?.objecter ms_handle_connect 0x7fa8a00a21c0 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969826283 0 0) 0x7fa8a0126fb0 con 0x7fa8a009fdf0 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): my global_id is 94284 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a0143070 con 0x7fa8a009fdf0 2024-09-18T11:52:05.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151045660 0 0) 0x7fa8a0142340 con 0x7fa8a00a21c0 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): my global_id is 18193 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a0126fb0 con 0x7fa8a00a21c0 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4211805633 0 0) 0x7fa884004f10 con 0x7fa8a00a01f0 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a0142340 con 0x7fa8a00a01f0 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa888001ce0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): dump: 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.950 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 369381966 0 0) 0x7fa8a0143070 con 0x7fa8a009fdf0 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa884004f10 con 0x7fa8a009fdf0 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa890001cc0 con 0x7fa8a009fdf0 2024-09-18T11:52:05.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): dump: 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 430264932 0 0) 0x7fa8a0126fb0 con 0x7fa8a00a21c0 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a0143070 con 0x7fa8a00a21c0 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8980020c0 con 0x7fa8a00a21c0 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient(hunting): dump: 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.953 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1752344093 0 0) 0x7fa8a0142340 con 0x7fa8a00a01f0 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6789/0 conn(0x7fa8a00a21c0 legacy=0x7fa8a014e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.112:6789/0 conn(0x7fa8a009fdf0 legacy=0x7fa8a0141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient: _finish_hunting 0 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 monclient: found mon.c 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a0126cd0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient: _finish_auth 0 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:05.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 5 monclient: authenticate success, global_id 104159 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 10 monclient: _renew_subs 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a0127290 con 0x7fa8a00a01f0 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 10 client.104159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 10 monclient: _renew_subs 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a0125fa0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8a5062700 1 librados: init done 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8880021b0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8880024c0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa89dd9b700 4 set_mon_vals no callback set 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.938+0000 7fa8977fe700 10 monclient: dump: 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8a5062700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa878006590 con 0x7fa878002840 2024-09-18T11:52:05.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa8880026d0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(342..342 src has 1..342) v4 ==== 5393+0+0 (unknown 1437224965 0 0) 0x7fa888043db0 con 0x7fa8a00a01f0 2024-09-18T11:52:05.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8977fe700 10 client.104159.objecter ms_dispatch 0x7fa8a0000fd0 osd_map(342..342 src has 1..342) v4 2024-09-18T11:52:05.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8977fe700 3 client.104159.objecter handle_osd_map got epochs [342,342] > 0 2024-09-18T11:52:05.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8977fe700 3 client.104159.objecter handle_osd_map decoding full epoch 342 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8977fe700 20 client.104159.objecter dump_active .. 0 homeless 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa89f59e700 10 monclient: get_auth_request con 0x7fa8840387a0 auth_method 0 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8977fe700 10 client.104159.objecter ms_handle_connect 0x7fa878002840 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa8977fe700 10 client.104159.objecter resend_mon_ops 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.942+0000 7fa89ed9d700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6789/0 conn(0x7fa878002840 legacy=0x7fa878004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264220466 0 0) 0x7fa878006590 con 0x7fa878002840 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 monclient(hunting): my global_id is 18196 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a0142340 con 0x7fa878002840 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3440489310 0 0) 0x7fa8a0142340 con 0x7fa878002840 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa878006590 con 0x7fa878002840 2024-09-18T11:52:05.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 client.104159.objecter ms_handle_connect 0x7fa8840387a0 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa890002290 con 0x7fa878002840 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 monclient: dump: 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.960 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.961 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3629424777 0 0) 0x7fa878006590 con 0x7fa878002840 2024-09-18T11:52:05.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa87800a140 con 0x7fa878002840 2024-09-18T11:52:05.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fa890005810 con 0x7fa878002840 2024-09-18T11:52:05.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:05.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:05.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6789/0 conn(0x7fa878002840 legacy=0x7fa878004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.946+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6789/0 conn(0x7fa878002840 legacy=0x7fa878004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:05.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.958+0000 7fa8a5062700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:52:05.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.958+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa878009670 con 0x7fa8780090e0 2024-09-18T11:52:05.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa89ed9d700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6789/0 conn(0x7fa8780090e0 legacy=0x7fa878008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:05.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 10 client.104159.objecter ms_handle_connect 0x7fa8780090e0 2024-09-18T11:52:05.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 10 client.104159.objecter resend_mon_ops 2024-09-18T11:52:05.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248913183 0 0) 0x7fa878009670 con 0x7fa8780090e0 2024-09-18T11:52:05.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:05.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 10 monclient(hunting): my global_id is 18199 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa878006590 con 0x7fa8780090e0 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 25333942 0 0) 0x7fa878006590 con 0x7fa8780090e0 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa878009670 con 0x7fa8780090e0 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8900029b0 con 0x7fa8780090e0 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 10 monclient: dump: 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:05.968 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 725447012 0 0) 0x7fa878009670 con 0x7fa8780090e0 2024-09-18T11:52:05.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.962+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa878009930 con 0x7fa8780090e0 2024-09-18T11:52:05.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 2539005073) 0x7fa890002cc0 con 0x7fa8780090e0 2024-09-18T11:52:05.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8977fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:52:05.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8977fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:05.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6789/0 conn(0x7fa8780090e0 legacy=0x7fa878008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8977fe700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6789/0 conn(0x7fa8780090e0 legacy=0x7fa878008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:05.970 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:52:05.970 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:05.970 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 36, 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:52:05.971 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:52:05.972 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:52:05.973 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:52:05.974 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.975 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.976 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.978 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:52:05.979 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:52:05.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:05.981 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:52:05.982 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:52:05.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 10 librados: watch_flush enter 2024-09-18T11:52:05.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 10 librados: watch_flush exit 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 20 client.104159.objecter shutdown clearing up homeless session... 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 10 client.104159.objecter successfully canceled tick 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.129:6812/20806 conn(0x7fa8840387a0 legacy=0x7fa88403ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 10 monclient: shutdown 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 >> v1:172.21.15.112:6790/0 conn(0x7fa8a00a01f0 legacy=0x7fa8a014ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa89f59e700 1 -- 172.21.15.112:0/1181152561 reap_dead start 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 shutdown_connections 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 >> 172.21.15.112:0/1181152561 conn(0x7fa8a00680d0 msgr2=0x7fa8a009c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 shutdown_connections 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 1 -- 172.21.15.112:0/1181152561 wait complete. 2024-09-18T11:52:05.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:05.966+0000 7fa8a5062700 1 librados: shutdown 2024-09-18T11:52:05.985 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:52:05.986 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T11:52:05.986 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-18T11:52:05.986 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T11:52:05.986 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T11:52:05.986 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T11:52:05.986 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:52:06.008 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T11:52:06.008 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T11:52:06.008 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T11:52:06.008 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T11:52:21.009 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T11:52:21.010 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T11:52:21.010 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T11:52:21.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:52:21.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.114+0000 7fbe963b6700 10 monclient: get_monmap_and_config 2024-09-18T11:52:21.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.114+0000 7fbe963b6700 10 monclient: build_initial_monmap 2024-09-18T11:52:21.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.114+0000 7fbe963b6700 10 monclient: monmap: 2024-09-18T11:52:21.120 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:21.120 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:21.120 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:21.120119+0000 2024-09-18T11:52:21.120 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:21.120119+0000 2024-09-18T11:52:21.121 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:21.121 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:21.121 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:21.121 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:21.121 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:21.121 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:21.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 1 Processor -- start 2024-09-18T11:52:21.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 1 -- start start 2024-09-18T11:52:21.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 10 monclient: init 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 10 monclient(hunting): picked mon.noname-a con 0x7fbe900a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 10 monclient(hunting): picked mon.noname-c con 0x7fbe9006c280 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 10 monclient(hunting): picked mon.noname-b con 0x7fbe9006c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9006e950 con 0x7fbe900a1ea0 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe900a2270 con 0x7fbe9006c280 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9006d130 con 0x7fbe9006c670 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe963b6700 10 monclient(hunting): authenticate will time out at 1312.961245s 2024-09-18T11:52:21.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe94bb3700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe90135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:21.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe95bb5700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fbe9006c670 0x7fbe901388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34768/0 (socket says 172.21.15.112:34768) 2024-09-18T11:52:21.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe95bb5700 1 -- 172.21.15.112:0/407242818 learned_addr learned my addr 172.21.15.112:0/407242818 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:21.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe953b4700 1 --1- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6789/0 conn(0x7fbe900a1ea0 0x7fbe9006bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38762/0 (socket says 172.21.15.112:38762) 2024-09-18T11:52:21.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe95bb5700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6790/0 conn(0x7fbe9006c670 legacy=0x7fbe901388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:21.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.118+0000 7fbe953b4700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6789/0 conn(0x7fbe900a1ea0 legacy=0x7fbe9006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:21.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.322+0000 7fbe94bb3700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe90135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:21.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:21.726+0000 7fbe94bb3700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe90135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:22.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:22.530+0000 7fbe94bb3700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe90135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 10 monclient(hunting): tick 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 1 monclient(hunting): continuing hunt 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6790/0 conn(0x7fbe9006c670 legacy=0x7fbe901388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe90135170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6789/0 conn(0x7fbe900a1ea0 legacy=0x7fbe9006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 10 monclient(hunting): picked mon.noname-a con 0x7fbe70003810 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 10 monclient(hunting): picked mon.noname-b con 0x7fbe9006c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 10 monclient(hunting): picked mon.noname-c con 0x7fbe9006c280 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 1 -- 172.21.15.112:0/407242818 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9009e370 con 0x7fbe70003810 2024-09-18T11:52:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 1 -- 172.21.15.112:0/407242818 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe900a2270 con 0x7fbe9006c280 2024-09-18T11:52:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 1 -- 172.21.15.112:0/407242818 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe70010530 con 0x7fbe9006c670 2024-09-18T11:52:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe95bb5700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe7000cd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.118+0000 7fbe8effd700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:24.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.122+0000 7fbe953b4700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6789/0 conn(0x7fbe70003810 legacy=0x7fbe70005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:24.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.122+0000 7fbe94bb3700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6790/0 conn(0x7fbe9006c670 legacy=0x7fbe70009500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:24.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.322+0000 7fbe95bb5700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe7000cd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:24.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:24.726+0000 7fbe95bb5700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe7000cd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:25.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:25.530+0000 7fbe95bb5700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe7000cd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255289343 0 0) 0x7fbe70010530 con 0x7fbe9006c670 2024-09-18T11:52:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient(hunting): my global_id is 104162 2024-09-18T11:52:26.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe7c003570 con 0x7fbe9006c670 2024-09-18T11:52:26.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1287116314 0 0) 0x7fbe7c003570 con 0x7fbe9006c670 2024-09-18T11:52:26.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.129:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe7000cd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:52:26.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6789/0 conn(0x7fbe70003810 legacy=0x7fbe70005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: _finish_hunting 0 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 monclient: found mon.noname-b 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe700117f0 con 0x7fbe9006c670 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: _finish_auth 0 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbe88004820 con 0x7fbe9006c670 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe963b6700 5 monclient: authenticate success, global_id 104162 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe963b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: dump: 2024-09-18T11:52:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe963b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe88004ab0 con 0x7fbe9006c670 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 1 -- 172.21.15.112:0/407242818 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbe88004e00 con 0x7fbe9006c670 2024-09-18T11:52:26.033 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe963b6700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe963b6700 4 set_mon_vals no callback set 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.026+0000 7fbe8ffff700 10 monclient: dump: 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.034 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 10 monclient: shutdown 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 -- 172.21.15.112:0/407242818 >> v1:172.21.15.112:6790/0 conn(0x7fbe9006c670 legacy=0x7fbe70009500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 -- 172.21.15.112:0/407242818 shutdown_connections 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 -- 172.21.15.112:0/407242818 >> 172.21.15.112:0/407242818 conn(0x7fbe90067da0 msgr2=0x7fbe900a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 -- 172.21.15.112:0/407242818 shutdown_connections 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 -- 172.21.15.112:0/407242818 wait complete. 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 10 monclient: build_initial_monmap 2024-09-18T11:52:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 10 monclient: monmap: 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 librados: starting msgr at 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 librados: starting objecter 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 Processor -- start 2024-09-18T11:52:26.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 -- start start 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 librados: setting wanted keys 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 1 librados: calling monclient init 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.030+0000 7fbe963b6700 10 monclient: init 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 10 monclient(hunting): picked mon.noname-b con 0x7fbe70003810 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 10 monclient(hunting): picked mon.noname-a con 0x7fbe9006c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 10 monclient(hunting): picked mon.noname-c con 0x7fbe9006c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9006e1a0 con 0x7fbe9006c280 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90140100 con 0x7fbe70003810 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9011f010 con 0x7fbe9006c670 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe963b6700 10 monclient(hunting): authenticate will time out at 1317.876725s 2024-09-18T11:52:26.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe953b4700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7fbe70003810 legacy=0x7fbe90131ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:26.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe95bb5700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fbe9006c670 0x7fbe90147cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34786/0 (socket says 172.21.15.112:34786) 2024-09-18T11:52:26.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe94bb3700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbe9006c280 0x7fbe901445a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38784/0 (socket says 172.21.15.112:38784) 2024-09-18T11:52:26.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe95bb5700 1 -- 172.21.15.112:0/944221318 learned_addr learned my addr 172.21.15.112:0/944221318 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:26.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe95bb5700 1 -- 172.21.15.112:0/944221318 >> v1:172.21.15.112:6790/0 conn(0x7fbe9006c670 legacy=0x7fbe90147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 10 client.?.objecter ms_handle_connect 0x7fbe9006c670 2024-09-18T11:52:26.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:26.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe94bb3700 1 -- 172.21.15.112:0/944221318 >> v1:172.21.15.112:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe901445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 10 client.?.objecter ms_handle_connect 0x7fbe9006c280 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527454643 0 0) 0x7fbe9011f010 con 0x7fbe9006c670 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 10 monclient(hunting): my global_id is 104165 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe70006cb0 con 0x7fbe9006c670 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.034+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 360171572 0 0) 0x7fbe70006cb0 con 0x7fbe9006c670 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe9011f010 con 0x7fbe9006c670 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbe80001c30 con 0x7fbe9006c670 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient(hunting): dump: 2024-09-18T11:52:26.043 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3895338469 0 0) 0x7fbe9011f010 con 0x7fbe9006c670 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 >> v1:172.21.15.129:6789/0 conn(0x7fbe70003810 legacy=0x7fbe90131ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 >> v1:172.21.15.112:6789/0 conn(0x7fbe9006c280 legacy=0x7fbe901445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient: _finish_hunting 0 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 monclient: found mon.c 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9011f2f0 con 0x7fbe9006c670 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient: _finish_auth 0 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 5 monclient: authenticate success, global_id 104165 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 10 monclient: _renew_subs 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9011d4e0 con 0x7fbe9006c670 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 10 client.104165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 10 monclient: _renew_subs 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9011e0b0 con 0x7fbe9006c670 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 1 librados: init done 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe80004be0 con 0x7fbe9006c670 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbe80004f70 con 0x7fbe9006c670 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8f7fe700 4 set_mon_vals no callback set 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 monclient: dump: 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.046 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fbe800050d0 con 0x7fbe9006c670 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe740050b0 con 0x7fbe9006c670 2024-09-18T11:52:26.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(342..342 src has 1..342) v4 ==== 5393+0+0 (unknown 1437224965 0 0) 0x7fbe80044000 con 0x7fbe9006c670 2024-09-18T11:52:26.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 10 client.104165.objecter ms_dispatch 0x7fbe90000fd0 osd_map(342..342 src has 1..342) v4 2024-09-18T11:52:26.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 3 client.104165.objecter handle_osd_map got epochs [342,342] > 0 2024-09-18T11:52:26.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 3 client.104165.objecter handle_osd_map decoding full epoch 342 2024-09-18T11:52:26.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe953b4700 10 monclient: get_auth_request con 0x7fbe70038a10 auth_method 0 2024-09-18T11:52:26.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.038+0000 7fbe8d7fa700 20 client.104165.objecter dump_active .. 0 homeless 2024-09-18T11:52:26.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.042+0000 7fbe8d7fa700 10 client.104165.objecter ms_handle_connect 0x7fbe70038a10 2024-09-18T11:52:26.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.050+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fbe80005380 con 0x7fbe9006c670 2024-09-18T11:52:26.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.050+0000 7fbe8d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:26.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.050+0000 7fbe8d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:26.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.270+0000 7fbe963b6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:52:26.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.270+0000 7fbe963b6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:52:26.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.270+0000 7fbe963b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.270+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbe740058e0 con 0x7fbe9006c670 2024-09-18T11:52:26.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.270+0000 7fbe8d7fa700 1 -- 172.21.15.112:0/944221318 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3500383647) 0x7fbe8001b980 con 0x7fbe9006c670 2024-09-18T11:52:26.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.270+0000 7fbe8d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:52:26.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.270+0000 7fbe8d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:26.276 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":38,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:52:26.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 10 librados: watch_flush enter 2024-09-18T11:52:26.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 10 librados: watch_flush exit 2024-09-18T11:52:26.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 20 client.104165.objecter shutdown clearing up homeless session... 2024-09-18T11:52:26.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 10 client.104165.objecter successfully canceled tick 2024-09-18T11:52:26.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 >> v1:172.21.15.129:6812/20806 conn(0x7fbe70038a10 legacy=0x7fbe7003aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 10 monclient: shutdown 2024-09-18T11:52:26.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:26.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 >> v1:172.21.15.112:6790/0 conn(0x7fbe9006c670 legacy=0x7fbe90147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 shutdown_connections 2024-09-18T11:52:26.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 >> 172.21.15.112:0/944221318 conn(0x7fbe90067da0 msgr2=0x7fbe9009c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:26.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 shutdown_connections 2024-09-18T11:52:26.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 1 -- 172.21.15.112:0/944221318 wait complete. 2024-09-18T11:52:26.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.274+0000 7fbe963b6700 1 librados: shutdown 2024-09-18T11:52:26.294 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T11:52:26.294 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T11:52:26.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:52:26.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient: get_monmap_and_config 2024-09-18T11:52:26.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient: build_initial_monmap 2024-09-18T11:52:26.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient: monmap: 2024-09-18T11:52:26.400 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:26.400 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.400 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:26.400571+0000 2024-09-18T11:52:26.400 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:26.400571+0000 2024-09-18T11:52:26.400 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:26.401 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.401 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:26.401 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:26.401 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:26.401 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 1 Processor -- start 2024-09-18T11:52:26.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 1 -- start start 2024-09-18T11:52:26.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient: init 2024-09-18T11:52:26.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:26.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:52:26.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f82b40a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:26.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f82b40a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f82b41383a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4096f90 con 0x7f82b40a0750 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40a1d90 con 0x7f82b40a0380 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4096cb0 con 0x7f82b41383a0 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b3fff700 10 monclient(hunting): authenticate will time out at 1318.241660s 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b2ffd700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f82b40a0380 legacy=0x7f82b4097970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b27fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f82b40a0750 0x7f82b40a0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38794/0 (socket says 172.21.15.112:38794) 2024-09-18T11:52:26.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.398+0000 7f82b27fc700 1 -- 172.21.15.112:0/1555922077 learned_addr learned my addr 172.21.15.112:0/1555922077 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b27fc700 1 -- 172.21.15.112:0/1555922077 >> v1:172.21.15.112:6789/0 conn(0x7f82b40a0750 legacy=0x7f82b40a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475882694 0 0) 0x7f82b4096f90 con 0x7f82b40a0750 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b37fe700 1 -- 172.21.15.112:0/1555922077 >> v1:172.21.15.112:6790/0 conn(0x7f82b41383a0 legacy=0x7f82b413a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient(hunting): my global_id is 94287 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c003570 con 0x7f82b40a0750 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3742936250 0 0) 0x7f829c003570 con 0x7f82b40a0750 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 >> v1:172.21.15.112:6790/0 conn(0x7f82b41383a0 legacy=0x7f82b413a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 >> v1:172.21.15.129:6789/0 conn(0x7f82b40a0380 legacy=0x7f82b4097970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:52:26.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 monclient: found mon.noname-a 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4097270 con 0x7f82b40a0750 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: _finish_auth 0 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f82a4001940 con 0x7f82b40a0750 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 5 monclient: authenticate success, global_id 94287 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: dump: 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.406 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a4001c10 con 0x7f82b40a0750 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 1 -- 172.21.15.112:0/1555922077 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f82a4004a90 con 0x7f82b40a0750 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b1ffb700 10 monclient: dump: 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:26.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 4 set_mon_vals no callback set 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 10 monclient: shutdown 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 1 -- 172.21.15.112:0/1555922077 >> v1:172.21.15.112:6789/0 conn(0x7f82b40a0750 legacy=0x7f82b40a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 1 -- 172.21.15.112:0/1555922077 shutdown_connections 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 1 -- 172.21.15.112:0/1555922077 >> 172.21.15.112:0/1555922077 conn(0x7f82b40a6410 msgr2=0x7f82b40a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 1 -- 172.21.15.112:0/1555922077 shutdown_connections 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 1 -- 172.21.15.112:0/1555922077 wait complete. 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 10 monclient: build_initial_monmap 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 10 monclient: monmap: 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:26.411 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 1 librados: starting msgr at 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.402+0000 7f82b3fff700 1 librados: starting objecter 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 1 Processor -- start 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 1 -- start start 2024-09-18T11:52:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 1 librados: setting wanted keys 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 1 librados: calling monclient init 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 10 monclient: init 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f82b40a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f82b40a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f82b41383a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40a2070 con 0x7f82b41383a0 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4099aa0 con 0x7f82b40a0750 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b409a990 con 0x7f82b40a0380 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b3fff700 10 monclient(hunting): authenticate will time out at 1318.249179s 2024-09-18T11:52:26.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b27fc700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f82b40a0750 legacy=0x7f82b41465e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:26.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b2ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f82b40a0380 0x7f82b409e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:34798/0 (socket says 172.21.15.112:34798) 2024-09-18T11:52:26.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b2ffd700 1 -- 172.21.15.112:0/3224451849 learned_addr learned my addr 172.21.15.112:0/3224451849 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:26.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b2ffd700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6790/0 conn(0x7f82b40a0380 legacy=0x7f82b409e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f82b37fe700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6789/0 conn(0x7f82b41383a0 legacy=0x7f82b4149d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 client.?.objecter ms_handle_connect 0x7f82b40a0380 2024-09-18T11:52:26.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:26.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 client.?.objecter ms_handle_connect 0x7f82b41383a0 2024-09-18T11:52:26.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:26.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218938438 0 0) 0x7f82b40a2070 con 0x7f82b41383a0 2024-09-18T11:52:26.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 monclient(hunting): my global_id is 94290 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a0004f10 con 0x7f82b41383a0 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701052818 0 0) 0x7f82b409a990 con 0x7f82b40a0380 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 10 monclient(hunting): my global_id is 104171 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.406+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b40a2070 con 0x7f82b40a0380 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2510725452 0 0) 0x7f82b40a2070 con 0x7f82b40a0380 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b409a990 con 0x7f82b40a0380 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f82a80044d0 con 0x7f82b40a0380 2024-09-18T11:52:26.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient(hunting): dump: 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:26.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1377440340 0 0) 0x7f82a0004f10 con 0x7f82b41383a0 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b40a2070 con 0x7f82b41383a0 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f82ac001f40 con 0x7f82b41383a0 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient(hunting): dump: 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.421 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2146829160 0 0) 0x7f82b409a990 con 0x7f82b40a0380 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.129:6789/0 conn(0x7f82b40a0750 legacy=0x7f82b41465e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6789/0 conn(0x7f82b41383a0 legacy=0x7f82b4149d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient: _finish_hunting 0 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 monclient: found mon.c 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b409ac70 con 0x7f82b40a0380 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient: _finish_auth 0 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:26.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 5 monclient: authenticate success, global_id 104171 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 10 monclient: _renew_subs 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b409af50 con 0x7f82b40a0380 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 10 client.104171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 10 monclient: _renew_subs 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b409f030 con 0x7f82b40a0380 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 1 librados: init done 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a8004800 con 0x7f82b40a0380 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f82a8004eb0 con 0x7f82b40a0380 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b17fa700 4 set_mon_vals no callback set 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 monclient: dump: 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f82a800f730 con 0x7f82b40a0380 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8294006590 con 0x7f8294002840 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(342..342 src has 1..342) v4 ==== 5393+0+0 (unknown 1437224965 0 0) 0x7f82a8043e60 con 0x7f82b40a0380 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 10 client.104171.objecter ms_dispatch 0x7f82b4000fd0 osd_map(342..342 src has 1..342) v4 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 3 client.104171.objecter handle_osd_map got epochs [342,342] > 0 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 3 client.104171.objecter handle_osd_map decoding full epoch 342 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f829affd700 20 client.104171.objecter dump_active .. 0 homeless 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.410+0000 7f82b37fe700 10 monclient: get_auth_request con 0x7f82a00382d0 auth_method 0 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f82b27fc700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6789/0 conn(0x7f8294002840 legacy=0x7f8294004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 client.104171.objecter ms_handle_connect 0x7f8294002840 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 client.104171.objecter resend_mon_ops 2024-09-18T11:52:26.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873097301 0 0) 0x7f8294006590 con 0x7f8294002840 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 monclient(hunting): my global_id is 94293 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b409a990 con 0x7f8294002840 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 24849057 0 0) 0x7f82b409a990 con 0x7f8294002840 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8294006590 con 0x7f8294002840 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f82a40015c0 con 0x7f8294002840 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 monclient: dump: 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.426 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 client.104171.objecter ms_handle_connect 0x7f82a00382d0 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1690325798 0 0) 0x7f8294006590 con 0x7f8294002840 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f829400a140 con 0x7f8294002840 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f82a4001990 con 0x7f8294002840 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:26.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6789/0 conn(0x7f8294002840 legacy=0x7f8294004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.414+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6789/0 conn(0x7f8294002840 legacy=0x7f8294004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:26.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.422+0000 7f82b3fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:52:26.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.422+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8294009650 con 0x7f82940090c0 2024-09-18T11:52:26.429 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f82b27fc700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6789/0 conn(0x7f82940090c0 legacy=0x7f8294008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 client.104171.objecter ms_handle_connect 0x7f82940090c0 2024-09-18T11:52:26.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 client.104171.objecter resend_mon_ops 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284705504 0 0) 0x7f8294009650 con 0x7f82940090c0 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 monclient(hunting): my global_id is 94296 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8294006590 con 0x7f82940090c0 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3050432395 0 0) 0x7f8294006590 con 0x7f82940090c0 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8294009650 con 0x7f82940090c0 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f82a40051f0 con 0x7f82940090c0 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:26.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 monclient: dump: 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 680478396 0 0) 0x7f8294009650 con 0x7f82940090c0 2024-09-18T11:52:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8294009910 con 0x7f82940090c0 2024-09-18T11:52:26.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4162 (unknown 0 0 2138217628) 0x7f82a4007390 con 0x7f82940090c0 2024-09-18T11:52:26.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:52:26.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:26.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6789/0 conn(0x7f82940090c0 legacy=0x7f8294008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.426+0000 7f829affd700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6789/0 conn(0x7f82940090c0 legacy=0x7f8294008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 38, 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:26.434 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:52:26.435 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:52:26.436 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:52:26.437 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.438 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:52:26.445 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:52:26.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 10 librados: watch_flush enter 2024-09-18T11:52:26.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 10 librados: watch_flush exit 2024-09-18T11:52:26.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 20 client.104171.objecter shutdown clearing up homeless session... 2024-09-18T11:52:26.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 10 client.104171.objecter successfully canceled tick 2024-09-18T11:52:26.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.129:6812/20806 conn(0x7f82a00382d0 legacy=0x7f82a003a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 10 monclient: shutdown 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 >> v1:172.21.15.112:6790/0 conn(0x7f82b40a0380 legacy=0x7f82b409e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b37fe700 1 -- 172.21.15.112:0/3224451849 reap_dead start 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b37fe700 1 -- 172.21.15.112:0/3224451849 reap_dead start 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 shutdown_connections 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 >> 172.21.15.112:0/3224451849 conn(0x7f82b40a6410 msgr2=0x7f82b41375c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 shutdown_connections 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 1 -- 172.21.15.112:0/3224451849 wait complete. 2024-09-18T11:52:26.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.430+0000 7f82b3fff700 1 librados: shutdown 2024-09-18T11:52:26.448 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:52:26.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T11:52:26.449 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:52:26.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.554+0000 7f6ebe22b700 10 monclient: get_monmap_and_config 2024-09-18T11:52:26.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.554+0000 7f6ebe22b700 10 monclient: build_initial_monmap 2024-09-18T11:52:26.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.554+0000 7f6ebe22b700 10 monclient: monmap: 2024-09-18T11:52:26.560 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:26.560 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.560 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:52:26.560344+0000 2024-09-18T11:52:26.560 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:52:26.560344+0000 2024-09-18T11:52:26.560 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:26.561 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.561 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:26.561 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:52:26.561 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:52:26.561 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 1 Processor -- start 2024-09-18T11:52:26.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 1 -- start start 2024-09-18T11:52:26.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 10 monclient: init 2024-09-18T11:52:26.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:26.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:52:26.562 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 10 monclient(hunting): picked mon.noname-c con 0x7f6eb80a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 10 monclient(hunting): picked mon.noname-b con 0x7f6eb806c280 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 10 monclient(hunting): picked mon.noname-a con 0x7f6eb806c670 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb806e950 con 0x7f6eb806c670 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb80a2270 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb806d130 con 0x7f6eb806c280 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebe22b700 10 monclient(hunting): authenticate will time out at 1318.401506s 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebd229700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6eb80a1ea0 legacy=0x7f6eb806bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebda2a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6eb806c670 0x7f6eb81388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38834/0 (socket says 172.21.15.112:38834) 2024-09-18T11:52:26.563 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.558+0000 7f6ebda2a700 1 -- 172.21.15.112:0/1031310349 learned_addr learned my addr 172.21.15.112:0/1031310349 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:26.565 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebda2a700 1 -- 172.21.15.112:0/1031310349 >> v1:172.21.15.112:6789/0 conn(0x7f6eb806c670 legacy=0x7f6eb81388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.565 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebca28700 1 -- 172.21.15.112:0/1031310349 >> v1:172.21.15.112:6790/0 conn(0x7f6eb806c280 legacy=0x7f6eb8135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782215287 0 0) 0x7f6eb806e950 con 0x7f6eb806c670 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient(hunting): my global_id is 94299 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea4003570 con 0x7f6eb806c670 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428989228 0 0) 0x7f6eb806d130 con 0x7f6eb806c280 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient(hunting): my global_id is 104174 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb806e950 con 0x7f6eb806c280 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132242568 0 0) 0x7f6ea4003570 con 0x7f6eb806c670 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 >> v1:172.21.15.112:6790/0 conn(0x7f6eb806c280 legacy=0x7f6eb8135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.566 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 >> v1:172.21.15.129:6789/0 conn(0x7f6eb80a1ea0 legacy=0x7f6eb806bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: _finish_hunting 0 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 monclient: found mon.noname-a 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb809e370 con 0x7f6eb806c670 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: _finish_auth 0 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6eb4001940 con 0x7f6eb806c670 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: dump: 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.567 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 5 monclient: authenticate success, global_id 94299 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6eb4001bd0 con 0x7f6eb806c670 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:26.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 1 -- 172.21.15.112:0/1031310349 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6eb4004a90 con 0x7f6eb806c670 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 10 monclient: get_monmap_and_config success 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 4 set_mon_vals no callback set 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6eabfff700 10 monclient: dump: 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.569 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.570 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.570 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 10 monclient: shutdown 2024-09-18T11:52:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 1 -- 172.21.15.112:0/1031310349 >> v1:172.21.15.112:6789/0 conn(0x7f6eb806c670 legacy=0x7f6eb81388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 1 -- 172.21.15.112:0/1031310349 shutdown_connections 2024-09-18T11:52:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 1 -- 172.21.15.112:0/1031310349 >> 172.21.15.112:0/1031310349 conn(0x7f6eb8067da0 msgr2=0x7f6eb80a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 1 -- 172.21.15.112:0/1031310349 shutdown_connections 2024-09-18T11:52:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 1 -- 172.21.15.112:0/1031310349 wait complete. 2024-09-18T11:52:26.575 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 10 monclient: build_initial_monmap 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.562+0000 7f6ebe22b700 10 monclient: monmap: 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 librados: starting msgr at 2024-09-18T11:52:26.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 librados: starting objecter 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 Processor -- start 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 -- start start 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 librados: setting wanted keys 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 librados: calling monclient init 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 10 monclient: init 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 10 monclient: _reopen_session rank -1 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 10 monclient(hunting): picked mon.noname-b con 0x7f6eb806c280 addr v1:172.21.15.129:6789/0 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 10 monclient(hunting): picked mon.noname-c con 0x7f6eb806c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 10 monclient(hunting): picked mon.noname-a con 0x7f6eb80a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb80a11c0 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb811c2b0 con 0x7f6eb806c280 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebd229700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6eb806c280 legacy=0x7f6eb81331b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:52:26.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb811c590 con 0x7f6eb806c670 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 10 monclient(hunting): _renew_subs 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebe22b700 10 monclient(hunting): authenticate will time out at 1318.410055s 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebda2a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6eb80a1ea0 0x7f6eb8149d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:38850/0 (socket says 172.21.15.112:38850) 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.566+0000 7f6ebda2a700 1 -- 172.21.15.112:0/2207627200 learned_addr learned my addr 172.21.15.112:0/2207627200 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebda2a700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6789/0 conn(0x7f6eb80a1ea0 legacy=0x7f6eb8149d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 client.?.objecter ms_handle_connect 0x7f6eb80a1ea0 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 client.?.objecter ms_handle_connect 0x7f6eb806c670 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebca28700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6790/0 conn(0x7f6eb806c670 legacy=0x7f6eb81465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580470366 0 0) 0x7f6eb80a11c0 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): my global_id is 94302 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e9c004f10 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277905899 0 0) 0x7f6eb811c590 con 0x7f6eb806c670 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): my global_id is 104177 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb80a11c0 con 0x7f6eb806c670 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1539006105 0 0) 0x7f6eb80a11c0 con 0x7f6eb806c670 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eb811c590 con 0x7f6eb806c670 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6eb0001d40 con 0x7f6eb806c670 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:52:26.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): dump: 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902289576 0 0) 0x7f6e9c004f10 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eb80a11c0 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6eb4001360 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient(hunting): dump: 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.581 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3475742752 0 0) 0x7f6eb80a11c0 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6790/0 conn(0x7f6eb806c670 legacy=0x7f6eb81465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.129:6789/0 conn(0x7f6eb806c280 legacy=0x7f6eb81331b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient: _finish_hunting 0 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 monclient: found mon.a 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb811c870 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient: _finish_auth 0 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 5 monclient: authenticate success, global_id 94302 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 10 monclient: _renew_subs 2024-09-18T11:52:26.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb8097fa0 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 10 client.94302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 10 monclient: _renew_subs 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb8133a60 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 1 librados: init done 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6eb4004d30 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6eb4005040 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 monclient: dump: 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.583 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6eaaffd700 4 set_mon_vals no callback set 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f6eb40051a0 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(342..342 src has 1..342) v4 ==== 5393+0+0 (unknown 1437224965 0 0) 0x7f6eb4043fa0 con 0x7f6eb80a1ea0 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 10 client.94302.objecter ms_dispatch 0x7f6eb8000fd0 osd_map(342..342 src has 1..342) v4 2024-09-18T11:52:26.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 3 client.94302.objecter handle_osd_map got epochs [342,342] > 0 2024-09-18T11:52:26.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 3 client.94302.objecter handle_osd_map decoding full epoch 342 2024-09-18T11:52:26.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ea97fa700 20 client.94302.objecter dump_active .. 0 homeless 2024-09-18T11:52:26.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:26.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e98006590 con 0x7f6e98002840 2024-09-18T11:52:26.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.570+0000 7f6ebd229700 10 monclient: get_auth_request con 0x7f6e9c0381c0 auth_method 0 2024-09-18T11:52:26.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 client.94302.objecter ms_handle_connect 0x7f6e98002840 2024-09-18T11:52:26.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ebca28700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6790/0 conn(0x7f6e98002840 legacy=0x7f6e98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 client.94302.objecter resend_mon_ops 2024-09-18T11:52:26.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 client.94302.objecter ms_handle_connect 0x7f6e9c0381c0 2024-09-18T11:52:26.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270431356 0 0) 0x7f6e98006590 con 0x7f6e98002840 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 monclient(hunting): my global_id is 104180 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb80a11c0 con 0x7f6e98002840 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3150546539 0 0) 0x7f6eb80a11c0 con 0x7f6e98002840 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e98006590 con 0x7f6e98002840 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6eb0002900 con 0x7f6e98002840 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 monclient: dump: 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.587 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2235480507 0 0) 0x7f6e98006590 con 0x7f6e98002840 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e9800a140 con 0x7f6e98002840 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6eb0005b90 con 0x7f6e98002840 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:52:26.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6790/0 conn(0x7f6e98002840 legacy=0x7f6e98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.574+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6790/0 conn(0x7f6e98002840 legacy=0x7f6e98004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:26.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.590+0000 7f6ebe22b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:52:26.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.590+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e98009670 con 0x7f6e980090e0 2024-09-18T11:52:26.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.590+0000 7f6ea97fa700 10 client.94302.objecter ms_handle_connect 0x7f6e980090e0 2024-09-18T11:52:26.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.590+0000 7f6ea97fa700 10 client.94302.objecter resend_mon_ops 2024-09-18T11:52:26.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.590+0000 7f6ebca28700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6790/0 conn(0x7f6e980090e0 legacy=0x7f6e98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:52:26.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683188740 0 0) 0x7f6e98009670 con 0x7f6e980090e0 2024-09-18T11:52:26.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 10 monclient(hunting): my global_id is 104183 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e98006590 con 0x7f6e980090e0 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3108812418 0 0) 0x7f6e98006590 con 0x7f6e980090e0 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e98009670 con 0x7f6e980090e0 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6eb0002eb0 con 0x7f6e980090e0 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 10 monclient: dump: 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.597 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:52:26.598 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:52:26.598 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:52:26.598 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:52:26.598 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:52:26.598 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:52:26.598 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:52:26.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3370770642 0 0) 0x7f6e98009670 con 0x7f6e980090e0 2024-09-18T11:52:26.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6e98009930 con 0x7f6e980090e0 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3990 (unknown 0 0 576402759) 0x7f6eb0002d30 con 0x7f6e980090e0 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6790/0 conn(0x7f6e980090e0 legacy=0x7f6e98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ea97fa700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6790/0 conn(0x7f6e980090e0 legacy=0x7f6e98008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:26.599 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 38, 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:26.600 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.601 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:52:26.602 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:52:26.603 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:52:26.604 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:26.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:26.606 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.608 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:26.609 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:52:26.610 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:52:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:52:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:52:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:52:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:52:26.611 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:52:26.611 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:52:26.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 10 librados: watch_flush enter 2024-09-18T11:52:26.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 10 librados: watch_flush exit 2024-09-18T11:52:26.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 20 client.94302.objecter shutdown clearing up homeless session... 2024-09-18T11:52:26.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 10 client.94302.objecter successfully canceled tick 2024-09-18T11:52:26.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.129:6812/20806 conn(0x7f6e9c0381c0 legacy=0x7f6e9c03a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 10 monclient: shutdown 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebda2a700 1 -- 172.21.15.112:0/2207627200 reap_dead start 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 >> v1:172.21.15.112:6789/0 conn(0x7f6eb80a1ea0 legacy=0x7f6eb8149d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 shutdown_connections 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 >> 172.21.15.112:0/2207627200 conn(0x7f6eb8067da0 msgr2=0x7f6eb809bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 shutdown_connections 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 1 -- 172.21.15.112:0/2207627200 wait complete. 2024-09-18T11:52:26.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:52:26.594+0000 7f6ebe22b700 1 librados: shutdown 2024-09-18T11:52:26.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:52:26.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T11:53:27.111 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-09-18T11:53:27.106+0000 7f5f70263700 -1 snap_mapper.add_oid found existing snaps mapped on 58:072b0a0f:test-rados-api-smithi129-23975-76::foo:28, removing 2024-09-18T11:53:27.112 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-09-18T11:53:27.106+0000 7f0314a41700 -1 snap_mapper.add_oid found existing snaps mapped on 58:072b0a0f:test-rados-api-smithi129-23975-76::foo:28, removing 2024-09-18T11:53:31.141 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-09-18T11:53:31.134+0000 7f5f70263700 -1 snap_mapper.add_oid found existing snaps mapped on 58:0772e8ad:test-rados-api-smithi129-23975-78::foo:29, removing 2024-09-18T11:53:31.141 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-09-18T11:53:31.134+0000 7f0314a41700 -1 snap_mapper.add_oid found existing snaps mapped on 58:0772e8ad:test-rados-api-smithi129-23975-78::foo:29, removing 2024-09-18T11:53:56.129 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-09-18T11:53:56.121+0000 7f031b24e700 -1 snap_mapper.add_oid found existing snaps mapped on 219:2f7dacb0:test-rados-api-smithi129-23975-80::foo:2, removing 2024-09-18T11:53:56.129 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-18T11:53:56.121+0000 7f7cc1d7c700 -1 snap_mapper.add_oid found existing snaps mapped on 219:2f7dacb0:test-rados-api-smithi129-23975-80::foo:2, removing 2024-09-18T11:53:56.615 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T11:53:56.615 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T11:53:56.615 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T11:53:56.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T11:53:56.618 INFO:tasks.ceph.mon.b:Started 2024-09-18T11:53:56.618 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T11:53:56.619 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T11:53:56.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T11:54:11.619 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T11:54:11.620 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T11:54:11.620 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:54:11.620 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:54:11.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient: get_monmap_and_config 2024-09-18T11:54:11.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient: build_initial_monmap 2024-09-18T11:54:11.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient: monmap: 2024-09-18T11:54:11.730 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:11.730 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:11.730373+0000 2024-09-18T11:54:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:11.730373+0000 2024-09-18T11:54:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:11.731 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 1 Processor -- start 2024-09-18T11:54:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 1 -- start start 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient: init 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient(hunting): picked mon.noname-c con 0x7f47ac06a8b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient(hunting): picked mon.noname-a con 0x7f47ac132330 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient(hunting): picked mon.noname-b con 0x7f47ac132e40 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac096ad0 con 0x7f47ac132330 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0a20e0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0a1740 con 0x7f47ac132e40 2024-09-18T11:54:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.727+0000 7f47b08f2700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47b08f2700 10 monclient(hunting): authenticate will time out at 1423.572013s 2024-09-18T11:54:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47aad9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f47ac132e40 0x7f47ac13a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35442/0 (socket says 172.21.15.112:35442) 2024-09-18T11:54:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47aad9d700 1 -- 172.21.15.112:0/968479099 learned_addr learned my addr 172.21.15.112:0/968479099 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:11.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a9d9b700 1 -- 172.21.15.112:0/968479099 >> v1:172.21.15.112:6789/0 conn(0x7f47ac132330 legacy=0x7f47ac132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:11.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47aad9d700 1 -- 172.21.15.112:0/968479099 >> v1:172.21.15.112:6790/0 conn(0x7f47ac132e40 legacy=0x7f47ac13a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:11.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47aa59c700 1 -- 172.21.15.112:0/968479099 >> v1:172.21.15.129:6789/0 conn(0x7f47ac06a8b0 legacy=0x7f47ac097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:11.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480808618 0 0) 0x7f47ac0a20e0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:11.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient(hunting): my global_id is 114100 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4794003570 con 0x7f47ac06a8b0 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 331798407 0 0) 0x7f4794003570 con 0x7f47ac06a8b0 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 >> v1:172.21.15.112:6790/0 conn(0x7f47ac132e40 legacy=0x7f47ac13a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 >> v1:172.21.15.112:6789/0 conn(0x7f47ac132330 legacy=0x7f47ac132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: _finish_hunting 0 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 monclient: found mon.noname-c 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac13ece0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: _finish_auth 0 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f479c001990 con 0x7f47ac06a8b0 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: dump: 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.738 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47b08f2700 5 monclient: authenticate success, global_id 114100 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47b08f2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f479c001c60 con 0x7f47ac06a8b0 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 1 -- 172.21.15.112:0/968479099 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f479c0020a0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47b08f2700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47b08f2700 4 set_mon_vals no callback set 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.731+0000 7f47a959a700 10 monclient: dump: 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:11.741 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:11.741 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:11.741 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:11.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 10 monclient: shutdown 2024-09-18T11:54:11.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:11.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 -- 172.21.15.112:0/968479099 >> v1:172.21.15.129:6789/0 conn(0x7f47ac06a8b0 legacy=0x7f47ac097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:11.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 -- 172.21.15.112:0/968479099 shutdown_connections 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 -- 172.21.15.112:0/968479099 >> 172.21.15.112:0/968479099 conn(0x7f47ac0a5b70 msgr2=0x7f47ac0a5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 -- 172.21.15.112:0/968479099 shutdown_connections 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 -- 172.21.15.112:0/968479099 wait complete. 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 10 monclient: build_initial_monmap 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 10 monclient: monmap: 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:11.744 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 librados: starting msgr at 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 librados: starting objecter 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 Processor -- start 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 -- start start 2024-09-18T11:54:11.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 librados: setting wanted keys 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 1 librados: calling monclient init 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.735+0000 7f47b08f2700 10 monclient: init 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 10 monclient(hunting): picked mon.noname-a con 0x7f47ac06a8b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 10 monclient(hunting): picked mon.noname-c con 0x7f47ac132330 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 10 monclient(hunting): picked mon.noname-b con 0x7f47ac132e40 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0a22e0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0a4170 con 0x7f47ac132e40 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac06dab0 con 0x7f47ac132330 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47b08f2700 10 monclient(hunting): authenticate will time out at 1423.580320s 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47aa59c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f47ac06a8b0 0x7f47ac06f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54068/0 (socket says 172.21.15.112:54068) 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47aa59c700 1 -- 172.21.15.112:0/703068844 learned_addr learned my addr 172.21.15.112:0/703068844 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47aa59c700 1 -- 172.21.15.112:0/703068844 >> v1:172.21.15.112:6789/0 conn(0x7f47ac06a8b0 legacy=0x7f47ac06f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f479a7fc700 10 client.?.objecter ms_handle_connect 0x7f47ac06a8b0 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f479a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47a9d9b700 1 -- 172.21.15.112:0/703068844 >> v1:172.21.15.112:6790/0 conn(0x7f47ac132330 legacy=0x7f47ac14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f479a7fc700 10 client.?.objecter ms_handle_connect 0x7f47ac132330 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f479a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f479a7fc700 10 client.?.objecter ms_handle_connect 0x7f47ac132e40 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f479a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.739+0000 7f47aad9d700 1 -- 172.21.15.112:0/703068844 >> v1:172.21.15.129:6789/0 conn(0x7f47ac132e40 legacy=0x7f47ac14e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723109929 0 0) 0x7f47ac0a22e0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 10 monclient(hunting): my global_id is 94305 2024-09-18T11:54:11.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f478c004f10 con 0x7f47ac06a8b0 2024-09-18T11:54:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4279134596 0 0) 0x7f478c004f10 con 0x7f47ac06a8b0 2024-09-18T11:54:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47ac0a22e0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f479c0020a0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 10 monclient(hunting): dump: 2024-09-18T11:54:11.761 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:11.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.759+0000 7f479a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222985238 0 0) 0x7f47ac06dab0 con 0x7f47ac132330 2024-09-18T11:54:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): my global_id is 104186 2024-09-18T11:54:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f478c004f10 con 0x7f47ac132330 2024-09-18T11:54:11.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968009933 0 0) 0x7f47ac0a4170 con 0x7f47ac132e40 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): my global_id is 114103 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47ac06dab0 con 0x7f47ac132e40 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3657261655 0 0) 0x7f47ac06dab0 con 0x7f47ac132e40 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47ac0a4170 con 0x7f47ac132e40 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f47a4001de0 con 0x7f47ac132e40 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient(hunting): dump: 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:11.770 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3372182451 0 0) 0x7f478c004f10 con 0x7f47ac132330 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47ac06dab0 con 0x7f47ac132330 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2156013693 0 0) 0x7f47ac0a22e0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 >> v1:172.21.15.112:6790/0 conn(0x7f47ac132330 legacy=0x7f47ac14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:11.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 >> v1:172.21.15.129:6789/0 conn(0x7f47ac132e40 legacy=0x7f47ac14e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 monclient: found mon.a 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac09e7d0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient: _finish_auth 0 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 5 monclient: authenticate success, global_id 94305 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 10 monclient: _renew_subs 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac06d790 con 0x7f47ac06a8b0 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 10 client.94305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:11.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 10 monclient: _renew_subs 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac06fc10 con 0x7f47ac06a8b0 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 1 librados: init done 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f479c002630 con 0x7f47ac06a8b0 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f479c001610 con 0x7f47ac06a8b0 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f479a7fc700 10 monclient: dump: 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:11.773 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47a8d99700 4 set_mon_vals no callback set 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.763+0000 7f47b08f2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f47b08f2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f47b08f2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47900050b0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f479c002980 con 0x7f47ac06a8b0 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(378..378 src has 1..378) v4 ==== 5008+0+0 (unknown 2197200371 0 0) 0x7f479c043ee0 con 0x7f47ac06a8b0 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f479a7fc700 10 client.94305.objecter ms_dispatch 0x7f47ac000fd0 osd_map(378..378 src has 1..378) v4 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f479a7fc700 3 client.94305.objecter handle_osd_map got epochs [378,378] > 0 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f479a7fc700 3 client.94305.objecter handle_osd_map decoding full epoch 378 2024-09-18T11:54:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f479a7fc700 20 client.94305.objecter dump_active .. 0 homeless 2024-09-18T11:54:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.767+0000 7f47a9d9b700 10 monclient: get_auth_request con 0x7f478c0384f0 auth_method 0 2024-09-18T11:54:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.771+0000 7f479a7fc700 10 client.94305.objecter ms_handle_connect 0x7f478c0384f0 2024-09-18T11:54:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.775+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f479c002c30 con 0x7f47ac06a8b0 2024-09-18T11:54:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.775+0000 7f479a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.775+0000 7f479a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:12.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:54:12.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:54:12.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f47900058e0 con 0x7f47ac06a8b0 2024-09-18T11:54:12.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f479a7fc700 1 -- 172.21.15.112:0/703068844 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3529253150) 0x7f479c014630 con 0x7f47ac06a8b0 2024-09-18T11:54:12.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f479a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:54:12.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f479a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:12.001 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 10 librados: watch_flush enter 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 10 librados: watch_flush exit 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 20 client.94305.objecter shutdown clearing up homeless session... 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 10 client.94305.objecter successfully canceled tick 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 >> v1:172.21.15.129:6812/20806 conn(0x7f478c0384f0 legacy=0x7f478c03a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 10 monclient: shutdown 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:11.999+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 >> v1:172.21.15.112:6789/0 conn(0x7f47ac06a8b0 legacy=0x7f47ac06f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.003+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 shutdown_connections 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.003+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 >> 172.21.15.112:0/703068844 conn(0x7f47ac0a5b70 msgr2=0x7f47ac0a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.003+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 shutdown_connections 2024-09-18T11:54:12.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.003+0000 7f47b08f2700 1 -- 172.21.15.112:0/703068844 wait complete. 2024-09-18T11:54:12.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.003+0000 7f47b08f2700 1 librados: shutdown 2024-09-18T11:54:12.014 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:54:12.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:54:12.015 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:54:12.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient: get_monmap_and_config 2024-09-18T11:54:12.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient: build_initial_monmap 2024-09-18T11:54:12.117 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient: monmap: 2024-09-18T11:54:12.117 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:12.117 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.117 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:12.117633+0000 2024-09-18T11:54:12.118 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:12.117633+0000 2024-09-18T11:54:12.118 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:12.118 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.118 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:12.118 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:12.119 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:12.119 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 1 Processor -- start 2024-09-18T11:54:12.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 1 -- start start 2024-09-18T11:54:12.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient: init 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient(hunting): picked mon.noname-c con 0x7f285006a780 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient(hunting): picked mon.noname-a con 0x7f285006d820 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient(hunting): picked mon.noname-b con 0x7f285009bcc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285009fe40 con 0x7f285006d820 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285006c970 con 0x7f285006a780 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285006b820 con 0x7f285009bcc0 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f28563e4700 10 monclient(hunting): authenticate will time out at 1423.959277s 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f2854981700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f285009bcc0 0x7f2850137c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35454/0 (socket says 172.21.15.112:35454) 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f2854981700 1 -- 172.21.15.112:0/3306720801 learned_addr learned my addr 172.21.15.112:0/3306720801 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:12.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f2847fff700 1 --1- 172.21.15.112:0/3306720801 >> v1:172.21.15.112:6789/0 conn(0x7f285006d820 0x7f2850134500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54078/0 (socket says 172.21.15.112:54078) 2024-09-18T11:54:12.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f284ffff700 1 -- 172.21.15.112:0/3306720801 >> v1:172.21.15.129:6789/0 conn(0x7f285006a780 legacy=0x7f285006ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T11:54:12.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f284ffff700 1 -- 172.21.15.112:0/3306720801 >> v1:172.21.15.129:6789/0 conn(0x7f285006a780 legacy=0x7f285006ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T11:54:12.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.115+0000 7f284ffff700 1 --1- 172.21.15.112:0/3306720801 >> v1:172.21.15.129:6789/0 conn(0x7f285006a780 0x7f285006ab50 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-18T11:54:12.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f2854981700 1 -- 172.21.15.112:0/3306720801 >> v1:172.21.15.112:6790/0 conn(0x7f285009bcc0 legacy=0x7f2850137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f2847fff700 1 -- 172.21.15.112:0/3306720801 >> v1:172.21.15.112:6789/0 conn(0x7f285006d820 legacy=0x7f2850134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476684658 0 0) 0x7f285006b820 con 0x7f285009bcc0 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient(hunting): my global_id is 104189 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c003570 con 0x7f285009bcc0 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309817394 0 0) 0x7f285009fe40 con 0x7f285006d820 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient(hunting): my global_id is 94308 2024-09-18T11:54:12.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f285006b820 con 0x7f285006d820 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 175353740 0 0) 0x7f283c003570 con 0x7f285009bcc0 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 >> v1:172.21.15.129:6789/0 conn(0x7f285006a780 legacy=0x7f285006ab50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 >> v1:172.21.15.112:6789/0 conn(0x7f285006d820 legacy=0x7f2850134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 monclient: found mon.noname-b 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285006c690 con 0x7f285009bcc0 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: _finish_auth 0 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 5 monclient: authenticate success, global_id 104189 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2848001cc0 con 0x7f285009bcc0 2024-09-18T11:54:12.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: dump: 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2848001f50 con 0x7f285009bcc0 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 4 set_mon_vals no callback set 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 1 -- 172.21.15.112:0/3306720801 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2848004e10 con 0x7f285009bcc0 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f284f7fe700 10 monclient: dump: 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.128 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 10 monclient: shutdown 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 1 -- 172.21.15.112:0/3306720801 >> v1:172.21.15.112:6790/0 conn(0x7f285009bcc0 legacy=0x7f2850137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 1 -- 172.21.15.112:0/3306720801 shutdown_connections 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 1 -- 172.21.15.112:0/3306720801 >> 172.21.15.112:0/3306720801 conn(0x7f28500680d0 msgr2=0x7f28500a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 1 -- 172.21.15.112:0/3306720801 shutdown_connections 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.119+0000 7f28563e4700 1 -- 172.21.15.112:0/3306720801 wait complete. 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient: build_initial_monmap 2024-09-18T11:54:12.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient: monmap: 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 librados: starting msgr at 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 librados: starting objecter 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 Processor -- start 2024-09-18T11:54:12.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 -- start start 2024-09-18T11:54:12.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 librados: setting wanted keys 2024-09-18T11:54:12.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 librados: calling monclient init 2024-09-18T11:54:12.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient: init 2024-09-18T11:54:12.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:12.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:12.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient(hunting): picked mon.noname-a con 0x7f285006a780 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:12.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient(hunting): picked mon.noname-b con 0x7f285006d820 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient(hunting): picked mon.noname-c con 0x7f285009bcc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285013b9d0 con 0x7f285006a780 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2850120920 con 0x7f285006d820 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2850120360 con 0x7f285009bcc0 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f28563e4700 10 monclient(hunting): authenticate will time out at 1423.966215s 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f284ffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f285006a780 0x7f285013b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54090/0 (socket says 172.21.15.112:54090) 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.123+0000 7f284ffff700 1 -- 172.21.15.112:0/3682916029 learned_addr learned my addr 172.21.15.112:0/3682916029 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284ffff700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6789/0 conn(0x7f285006a780 legacy=0x7f285013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 client.?.objecter ms_handle_connect 0x7f285006a780 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 client.?.objecter ms_handle_connect 0x7f285009bcc0 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f2854981700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6790/0 conn(0x7f285009bcc0 legacy=0x7f2850147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858442417 0 0) 0x7f285013b9d0 con 0x7f285006a780 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): my global_id is 94311 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f2847fff700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.129:6789/0 conn(0x7f285006d820 legacy=0x7f2850144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2830004f10 con 0x7f285006a780 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 client.?.objecter ms_handle_connect 0x7f285006d820 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f2854981700 1 --1- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6790/0 conn(0x7f285009bcc0 0x7f2850147c40 :-1 s=OPENED pgs=9013 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f2850120360 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668042163 0 0) 0x7f2850120920 con 0x7f285006d820 2024-09-18T11:54:12.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): my global_id is 114106 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f285013b9d0 con 0x7f285006d820 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 653285590 0 0) 0x7f2830004f10 con 0x7f285006a780 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2850120920 con 0x7f285006a780 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2840002cf0 con 0x7f285006a780 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): dump: 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.139 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3990090946 0 0) 0x7f285013b9d0 con 0x7f285006d820 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2830004f10 con 0x7f285006d820 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 402776891 0 0) 0x7f2850120920 con 0x7f285006a780 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6790/0 conn(0x7f285009bcc0 legacy=0x7f2850147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.129:6789/0 conn(0x7f285006d820 legacy=0x7f2850144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient: _finish_hunting 0 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 monclient: found mon.a 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850120640 con 0x7f285006a780 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient: _finish_auth 0 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 5 monclient: authenticate success, global_id 94311 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221918499 0 0) 0x7f2850120360 con 0x7f285009bcc0 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 10 monclient: _renew_subs 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28500a0330 con 0x7f285006a780 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:54:12.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 10 client.94311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 10 monclient: _renew_subs 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f285013c280 con 0x7f285006a780 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f28563e4700 1 librados: init done 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2840001280 con 0x7f285006a780 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2840001590 con 0x7f285006a780 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284effd700 4 set_mon_vals no callback set 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 10 monclient: dump: 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.127+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f28400017a0 con 0x7f285006a780 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f28563e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2834006590 con 0x7f2834002840 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(378..378 src has 1..378) v4 ==== 5008+0+0 (unknown 2197200371 0 0) 0x7f2840043cf0 con 0x7f285006a780 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 client.94311.objecter ms_dispatch 0x7f2850000fd0 osd_map(378..378 src has 1..378) v4 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 3 client.94311.objecter handle_osd_map got epochs [378,378] > 0 2024-09-18T11:54:12.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 3 client.94311.objecter handle_osd_map decoding full epoch 378 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 20 client.94311.objecter dump_active .. 0 homeless 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f2854981700 10 monclient: get_auth_request con 0x7f2830038230 auth_method 0 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 client.94311.objecter ms_handle_connect 0x7f2834002840 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 client.94311.objecter resend_mon_ops 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f2847fff700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6789/0 conn(0x7f2834002840 legacy=0x7f2834004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572583697 0 0) 0x7f2834006590 con 0x7f2834002840 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 monclient(hunting): my global_id is 94314 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2850120360 con 0x7f2834002840 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 345682880 0 0) 0x7f2850120360 con 0x7f2834002840 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2834006590 con 0x7f2834002840 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f28380019f0 con 0x7f2834002840 2024-09-18T11:54:12.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 monclient: dump: 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 10 client.94311.objecter ms_handle_connect 0x7f2830038230 2024-09-18T11:54:12.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1590868977 0 0) 0x7f2834006590 con 0x7f2834002840 2024-09-18T11:54:12.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.131+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f283400a140 con 0x7f2834002840 2024-09-18T11:54:12.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.135+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2838006fc0 con 0x7f2834002840 2024-09-18T11:54:12.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.135+0000 7f284cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.135+0000 7f284cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:12.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.135+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6789/0 conn(0x7f2834002840 legacy=0x7f2834004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.135+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6789/0 conn(0x7f2834002840 legacy=0x7f2834004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:12.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f28563e4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:54:12.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2834009f10 con 0x7f2834006900 2024-09-18T11:54:12.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f2847fff700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6789/0 conn(0x7f2834006900 legacy=0x7f2834006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f284cff9700 10 client.94311.objecter ms_handle_connect 0x7f2834006900 2024-09-18T11:54:12.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f284cff9700 10 client.94311.objecter resend_mon_ops 2024-09-18T11:54:12.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530058853 0 0) 0x7f2834009f10 con 0x7f2834006900 2024-09-18T11:54:12.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f284cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f284cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f284cff9700 10 monclient(hunting): my global_id is 94317 2024-09-18T11:54:12.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.143+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2834006590 con 0x7f2834006900 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 569236900 0 0) 0x7f2834006590 con 0x7f2834006900 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2834009f10 con 0x7f2834006900 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f2838005470 con 0x7f2834006900 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 10 monclient: dump: 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.149 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2035110715 0 0) 0x7f2834009f10 con 0x7f2834006900 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2834008760 con 0x7f2834006900 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4173 (unknown 0 0 2537810715) 0x7f2838006800 con 0x7f2834006900 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6789/0 conn(0x7f2834006900 legacy=0x7f2834006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f284cff9700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6789/0 conn(0x7f2834006900 legacy=0x7f2834006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 46, 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:54:12.151 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.152 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.153 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:54:12.154 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.155 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.156 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:12.157 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.158 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.160 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.161 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 6 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.163 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:54:12.163 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 10 librados: watch_flush enter 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 10 librados: watch_flush exit 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 20 client.94311.objecter shutdown clearing up homeless session... 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 10 client.94311.objecter successfully canceled tick 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.129:6812/20806 conn(0x7f2830038230 legacy=0x7f283003a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 10 monclient: shutdown 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 >> v1:172.21.15.112:6789/0 conn(0x7f285006a780 legacy=0x7f285013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f2854981700 1 -- 172.21.15.112:0/3682916029 reap_dead start 2024-09-18T11:54:12.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f2854981700 1 -- 172.21.15.112:0/3682916029 reap_dead start 2024-09-18T11:54:12.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 shutdown_connections 2024-09-18T11:54:12.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 >> 172.21.15.112:0/3682916029 conn(0x7f28500680d0 msgr2=0x7f285009c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 shutdown_connections 2024-09-18T11:54:12.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 1 -- 172.21.15.112:0/3682916029 wait complete. 2024-09-18T11:54:12.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.147+0000 7f28563e4700 1 librados: shutdown 2024-09-18T11:54:12.166 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:54:12.166 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:54:12.166 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:54:12.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient: get_monmap_and_config 2024-09-18T11:54:12.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient: build_initial_monmap 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient: monmap: 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:12.281094+0000 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:12.281094+0000 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:12.281 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 1 Processor -- start 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 1 -- start start 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient: init 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient(hunting): picked mon.noname-a con 0x7f10200a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient(hunting): picked mon.noname-c con 0x7f10200a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient(hunting): picked mon.noname-b con 0x7f10201383a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1020096f90 con 0x7f10200a0380 2024-09-18T11:54:12.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10200a1d90 con 0x7f10200a0750 2024-09-18T11:54:12.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1020096cb0 con 0x7f10201383a0 2024-09-18T11:54:12.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:12.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10267bc700 10 monclient(hunting): authenticate will time out at 1424.122320s 2024-09-18T11:54:12.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f1025fbb700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f10201383a0 0x7f102013a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35478/0 (socket says 172.21.15.112:35478) 2024-09-18T11:54:12.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f10257ba700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f10200a0380 0x7f1020097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54114/0 (socket says 172.21.15.112:54114) 2024-09-18T11:54:12.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.279+0000 7f1025fbb700 1 -- 172.21.15.112:0/3025210825 learned_addr learned my addr 172.21.15.112:0/3025210825 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:12.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f1025fbb700 1 -- 172.21.15.112:0/3025210825 >> v1:172.21.15.112:6790/0 conn(0x7f10201383a0 legacy=0x7f102013a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10257ba700 1 -- 172.21.15.112:0/3025210825 >> v1:172.21.15.112:6789/0 conn(0x7f10200a0380 legacy=0x7f1020097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924364555 0 0) 0x7f1020096cb0 con 0x7f10201383a0 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): my global_id is 104195 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f1024fb9700 1 -- 172.21.15.112:0/3025210825 >> v1:172.21.15.129:6789/0 conn(0x7f10200a0750 legacy=0x7f10200a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1008003570 con 0x7f10201383a0 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152408730 0 0) 0x7f1020096f90 con 0x7f10200a0380 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): my global_id is 94320 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1020096cb0 con 0x7f10200a0380 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172962431 0 0) 0x7f10200a1d90 con 0x7f10200a0750 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient(hunting): my global_id is 114109 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1020096f90 con 0x7f10200a0750 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4072562039 0 0) 0x7f1020096cb0 con 0x7f10200a0380 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 >> v1:172.21.15.112:6790/0 conn(0x7f10201383a0 legacy=0x7f102013a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 >> v1:172.21.15.129:6789/0 conn(0x7f10200a0750 legacy=0x7f10200a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: _finish_hunting 0 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 monclient: found mon.noname-a 2024-09-18T11:54:12.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1020097270 con 0x7f10200a0380 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: _finish_auth 0 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 5 monclient: authenticate success, global_id 94320 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f10140019e0 con 0x7f10200a0380 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: dump: 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.289 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1014001b40 con 0x7f10200a0380 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 1 -- 172.21.15.112:0/3025210825 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1014004830 con 0x7f10200a0380 2024-09-18T11:54:12.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 4 set_mon_vals no callback set 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f101ffff700 10 monclient: dump: 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 10 monclient: shutdown 2024-09-18T11:54:12.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 1 -- 172.21.15.112:0/3025210825 >> v1:172.21.15.112:6789/0 conn(0x7f10200a0380 legacy=0x7f1020097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 1 -- 172.21.15.112:0/3025210825 shutdown_connections 2024-09-18T11:54:12.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 1 -- 172.21.15.112:0/3025210825 >> 172.21.15.112:0/3025210825 conn(0x7f10200a6410 msgr2=0x7f10200a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 1 -- 172.21.15.112:0/3025210825 shutdown_connections 2024-09-18T11:54:12.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.283+0000 7f10267bc700 1 -- 172.21.15.112:0/3025210825 wait complete. 2024-09-18T11:54:12.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient: build_initial_monmap 2024-09-18T11:54:12.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient: monmap: 2024-09-18T11:54:12.297 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:12.297 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:12.297 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 librados: starting msgr at 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 librados: starting objecter 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 Processor -- start 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 -- start start 2024-09-18T11:54:12.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 librados: setting wanted keys 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 librados: calling monclient init 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient: init 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient(hunting): picked mon.noname-a con 0x7f10200a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient(hunting): picked mon.noname-b con 0x7f10200a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient(hunting): picked mon.noname-c con 0x7f10201383a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10200a2070 con 0x7f10200a0380 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102013b050 con 0x7f10200a0750 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102009ad20 con 0x7f10201383a0 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10267bc700 10 monclient(hunting): authenticate will time out at 1424.130368s 2024-09-18T11:54:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10257ba700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f10200a0380 0x7f102009e830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54128/0 (socket says 172.21.15.112:54128) 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f1025fbb700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f10201383a0 0x7f1020149e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35486/0 (socket says 172.21.15.112:35486) 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.287+0000 7f10257ba700 1 -- 172.21.15.112:0/3384378526 learned_addr learned my addr 172.21.15.112:0/3384378526 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f1025fbb700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6790/0 conn(0x7f10201383a0 legacy=0x7f1020149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 client.?.objecter ms_handle_connect 0x7f10201383a0 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10257ba700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6789/0 conn(0x7f10200a0380 legacy=0x7f102009e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 client.?.objecter ms_handle_connect 0x7f10200a0380 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297824918 0 0) 0x7f102009ad20 con 0x7f10201383a0 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): my global_id is 104198 2024-09-18T11:54:12.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f1024fb9700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.129:6789/0 conn(0x7f10200a0750 legacy=0x7f1020146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f100c004f10 con 0x7f10201383a0 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 client.?.objecter ms_handle_connect 0x7f10200a0750 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892747417 0 0) 0x7f10200a2070 con 0x7f10200a0380 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): my global_id is 94323 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102009ad20 con 0x7f10200a0380 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491515340 0 0) 0x7f102013b050 con 0x7f10200a0750 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): my global_id is 114112 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10200a2070 con 0x7f10200a0750 2024-09-18T11:54:12.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4171506336 0 0) 0x7f100c004f10 con 0x7f10201383a0 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f102013b050 con 0x7f10201383a0 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1018001cc0 con 0x7f10201383a0 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): dump: 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.302 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3631509748 0 0) 0x7f102009ad20 con 0x7f10200a0380 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f100c004f10 con 0x7f10200a0380 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1014001cb0 con 0x7f10200a0380 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient(hunting): dump: 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.303 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1750800964 0 0) 0x7f102013b050 con 0x7f10201383a0 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.129:6789/0 conn(0x7f10200a0750 legacy=0x7f1020146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6789/0 conn(0x7f10200a0380 legacy=0x7f102009e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient: _finish_hunting 0 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 monclient: found mon.c 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:12.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102009b2e0 con 0x7f10201383a0 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient: _finish_auth 0 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 5 monclient: authenticate success, global_id 104198 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 10 monclient: _renew_subs 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102009aa40 con 0x7f10201383a0 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 10 client.104198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 10 monclient: _renew_subs 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10201418e0 con 0x7f10201383a0 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 1 librados: init done 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1018004ae0 con 0x7f10201383a0 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1018004df0 con 0x7f10201383a0 2024-09-18T11:54:12.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 10 monclient: dump: 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101f7fe700 4 set_mon_vals no callback set 2024-09-18T11:54:12.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f101800f670 con 0x7f10201383a0 2024-09-18T11:54:12.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.291+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1000006590 con 0x7f1000002840 2024-09-18T11:54:12.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(378..378 src has 1..378) v4 ==== 5008+0+0 (unknown 2197200371 0 0) 0x7f1018043bc0 con 0x7f10201383a0 2024-09-18T11:54:12.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 client.104198.objecter ms_dispatch 0x7f1020000fd0 osd_map(378..378 src has 1..378) v4 2024-09-18T11:54:12.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 3 client.104198.objecter handle_osd_map got epochs [378,378] > 0 2024-09-18T11:54:12.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 3 client.104198.objecter handle_osd_map decoding full epoch 378 2024-09-18T11:54:12.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 20 client.104198.objecter dump_active .. 0 homeless 2024-09-18T11:54:12.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f1024fb9700 10 monclient: get_auth_request con 0x7f100c038440 auth_method 0 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 client.104198.objecter ms_handle_connect 0x7f1000002840 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f10257ba700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6790/0 conn(0x7f1000002840 legacy=0x7f1000004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 client.104198.objecter resend_mon_ops 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362073173 0 0) 0x7f1000006590 con 0x7f1000002840 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 monclient(hunting): my global_id is 104201 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102013b050 con 0x7f1000002840 2024-09-18T11:54:12.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1556249554 0 0) 0x7f102013b050 con 0x7f1000002840 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1000006590 con 0x7f1000002840 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 client.104198.objecter ms_handle_connect 0x7f100c038440 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1014005180 con 0x7f1000002840 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 monclient: dump: 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.310 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1823441118 0 0) 0x7f1000006590 con 0x7f1000002840 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f100000a140 con 0x7f1000002840 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f1014007340 con 0x7f1000002840 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6790/0 conn(0x7f1000002840 legacy=0x7f1000004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.295+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6790/0 conn(0x7f1000002840 legacy=0x7f1000004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f10267bc700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:54:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1000007550 con 0x7f1000006900 2024-09-18T11:54:12.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 10 client.104198.objecter ms_handle_connect 0x7f1000006900 2024-09-18T11:54:12.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 10 client.104198.objecter resend_mon_ops 2024-09-18T11:54:12.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f10257ba700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6790/0 conn(0x7f1000006900 legacy=0x7f1000006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752053319 0 0) 0x7f1000007550 con 0x7f1000006900 2024-09-18T11:54:12.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 10 monclient(hunting): my global_id is 104204 2024-09-18T11:54:12.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1000006590 con 0x7f1000006900 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1663294426 0 0) 0x7f1000006590 con 0x7f1000006900 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1000007550 con 0x7f1000006900 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f10140058e0 con 0x7f1000006900 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.307+0000 7f101d7fa700 10 monclient: dump: 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.314 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3186396516 0 0) 0x7f1000007550 con 0x7f1000006900 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10000087b0 con 0x7f1000006900 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 1008646084) 0x7f1014007570 con 0x7f1000006900 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f101d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f101d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6790/0 conn(0x7f1000006900 legacy=0x7f1000006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f101d7fa700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6790/0 conn(0x7f1000006900 legacy=0x7f1000006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 46, 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:54:12.316 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.317 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.318 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:54:12.319 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.320 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.321 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.322 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.324 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.325 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.326 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:54:12.327 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 10 librados: watch_flush enter 2024-09-18T11:54:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 10 librados: watch_flush exit 2024-09-18T11:54:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 20 client.104198.objecter shutdown clearing up homeless session... 2024-09-18T11:54:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 10 client.104198.objecter successfully canceled tick 2024-09-18T11:54:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.129:6812/20806 conn(0x7f100c038440 legacy=0x7f100c03a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 10 monclient: shutdown 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 >> v1:172.21.15.112:6790/0 conn(0x7f10201383a0 legacy=0x7f1020149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f1025fbb700 1 -- 172.21.15.112:0/3384378526 reap_dead start 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f1025fbb700 1 -- 172.21.15.112:0/3384378526 reap_dead start 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 shutdown_connections 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 >> 172.21.15.112:0/3384378526 conn(0x7f10200a6410 msgr2=0x7f10200a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 shutdown_connections 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 1 -- 172.21.15.112:0/3384378526 wait complete. 2024-09-18T11:54:12.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.311+0000 7f10267bc700 1 librados: shutdown 2024-09-18T11:54:12.330 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:54:12.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:54:12.331 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:54:12.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.439+0000 7f5c5f620700 10 monclient: get_monmap_and_config 2024-09-18T11:54:12.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.439+0000 7f5c5f620700 10 monclient: build_initial_monmap 2024-09-18T11:54:12.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.439+0000 7f5c5f620700 10 monclient: monmap: 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:12.443990+0000 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:12.443990+0000 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:12.444 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 1 Processor -- start 2024-09-18T11:54:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 1 -- start start 2024-09-18T11:54:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 10 monclient: init 2024-09-18T11:54:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 10 monclient(hunting): picked mon.noname-c con 0x7f5c58130420 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 10 monclient(hunting): picked mon.noname-a con 0x7f5c5806ea60 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 10 monclient(hunting): picked mon.noname-b con 0x7f5c58136aa0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c58130f00 con 0x7f5c5806ea60 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5806c990 con 0x7f5c58130420 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c580a12c0 con 0x7f5c58136aa0 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5f620700 10 monclient(hunting): authenticate will time out at 1424.285534s 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5cbbb700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5c5806ea60 0x7f5c58135380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54142/0 (socket says 172.21.15.112:54142) 2024-09-18T11:54:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5cbbb700 1 -- 172.21.15.112:0/1750949793 learned_addr learned my addr 172.21.15.112:0/1750949793 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:12.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.443+0000 7f5c5cbbb700 1 -- 172.21.15.112:0/1750949793 >> v1:172.21.15.112:6789/0 conn(0x7f5c5806ea60 legacy=0x7f5c58135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5dbbd700 1 -- 172.21.15.112:0/1750949793 >> v1:172.21.15.112:6790/0 conn(0x7f5c58136aa0 legacy=0x7f5c58138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5d3bc700 1 -- 172.21.15.112:0/1750949793 >> v1:172.21.15.129:6789/0 conn(0x7f5c58130420 legacy=0x7f5c581307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318278814 0 0) 0x7f5c580a12c0 con 0x7f5c58136aa0 2024-09-18T11:54:12.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): my global_id is 104207 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c44003570 con 0x7f5c58136aa0 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141121127 0 0) 0x7f5c58130f00 con 0x7f5c5806ea60 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): my global_id is 94326 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c580a12c0 con 0x7f5c5806ea60 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035401974 0 0) 0x7f5c5806c990 con 0x7f5c58130420 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient(hunting): my global_id is 114115 2024-09-18T11:54:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c58130f00 con 0x7f5c58130420 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 283588317 0 0) 0x7f5c44003570 con 0x7f5c58136aa0 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 >> v1:172.21.15.129:6789/0 conn(0x7f5c58130420 legacy=0x7f5c581307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 >> v1:172.21.15.112:6789/0 conn(0x7f5c5806ea60 legacy=0x7f5c58135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: _finish_hunting 0 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 monclient: found mon.noname-b 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5806d7d0 con 0x7f5c58136aa0 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: _finish_auth 0 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 5 monclient: authenticate success, global_id 104207 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c50001d10 con 0x7f5c58136aa0 2024-09-18T11:54:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: dump: 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.454 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c50001fa0 con 0x7f5c58136aa0 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 4 set_mon_vals no callback set 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 1 -- 172.21.15.112:0/1750949793 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c50002420 con 0x7f5c58136aa0 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c4bfff700 10 monclient: dump: 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.455 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 10 monclient: shutdown 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 -- 172.21.15.112:0/1750949793 >> v1:172.21.15.112:6790/0 conn(0x7f5c58136aa0 legacy=0x7f5c58138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 -- 172.21.15.112:0/1750949793 shutdown_connections 2024-09-18T11:54:12.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 -- 172.21.15.112:0/1750949793 >> 172.21.15.112:0/1750949793 conn(0x7f5c580680d0 msgr2=0x7f5c580a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 -- 172.21.15.112:0/1750949793 shutdown_connections 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 -- 172.21.15.112:0/1750949793 wait complete. 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 10 monclient: build_initial_monmap 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 10 monclient: monmap: 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:12.457 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 librados: starting msgr at 2024-09-18T11:54:12.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 librados: starting objecter 2024-09-18T11:54:12.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 Processor -- start 2024-09-18T11:54:12.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 -- start start 2024-09-18T11:54:12.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 librados: setting wanted keys 2024-09-18T11:54:12.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 1 librados: calling monclient init 2024-09-18T11:54:12.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.447+0000 7f5c5f620700 10 monclient: init 2024-09-18T11:54:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 10 monclient(hunting): picked mon.noname-a con 0x7f5c58130420 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 10 monclient(hunting): picked mon.noname-b con 0x7f5c58136aa0 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 10 monclient(hunting): picked mon.noname-c con 0x7f5c5811acb0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5811a660 con 0x7f5c58130420 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5809bce0 con 0x7f5c58136aa0 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c58136840 con 0x7f5c5811acb0 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5f620700 10 monclient(hunting): authenticate will time out at 1424.292708s 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5d3bc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5c58130420 0x7f5c58118660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54156/0 (socket says 172.21.15.112:54156) 2024-09-18T11:54:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5d3bc700 1 -- 172.21.15.112:0/3718338763 learned_addr learned my addr 172.21.15.112:0/3718338763 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5dbbd700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.112:6790/0 conn(0x7f5c5811acb0 legacy=0x7f5c58119480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 client.?.objecter ms_handle_connect 0x7f5c5811acb0 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 client.?.objecter ms_handle_connect 0x7f5c58130420 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5d3bc700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.112:6789/0 conn(0x7f5c58130420 legacy=0x7f5c58118660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584916432 0 0) 0x7f5c58136840 con 0x7f5c5811acb0 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): my global_id is 104210 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c3c004f40 con 0x7f5c5811acb0 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c5cbbb700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6789/0 conn(0x7f5c58136aa0 legacy=0x7f5c58118d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997981121 0 0) 0x7f5c5811a660 con 0x7f5c58130420 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): my global_id is 94329 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c58136840 con 0x7f5c58130420 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 client.?.objecter ms_handle_connect 0x7f5c58136aa0 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058522285 0 0) 0x7f5c3c004f40 con 0x7f5c5811acb0 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c5811a660 con 0x7f5c5811acb0 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c50002420 con 0x7f5c5811acb0 2024-09-18T11:54:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): dump: 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925442584 0 0) 0x7f5c5809bce0 con 0x7f5c58136aa0 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 10 monclient(hunting): my global_id is 114118 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.451+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c3c004f40 con 0x7f5c58136aa0 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 391593329 0 0) 0x7f5c58136840 con 0x7f5c58130420 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c5809bce0 con 0x7f5c58130420 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c4c001c30 con 0x7f5c58130420 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient(hunting): dump: 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 76780415 0 0) 0x7f5c5811a660 con 0x7f5c5811acb0 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6789/0 conn(0x7f5c58136aa0 legacy=0x7f5c58118d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.112:6789/0 conn(0x7f5c58130420 legacy=0x7f5c58118660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient: _finish_hunting 0 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 monclient: found mon.c 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5811bba0 con 0x7f5c5811acb0 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient: _finish_auth 0 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 5 monclient: authenticate success, global_id 104210 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 10 monclient: _renew_subs 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5811a0e0 con 0x7f5c5811acb0 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 10 client.104210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 10 monclient: _renew_subs 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5814b120 con 0x7f5c5811acb0 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 1 librados: init done 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c50001950 con 0x7f5c5811acb0 2024-09-18T11:54:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c50002a70 con 0x7f5c5811acb0 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c4b7fe700 4 set_mon_vals no callback set 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 monclient: dump: 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.471 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5c38006590 con 0x7f5c38002840 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5c50002bd0 con 0x7f5c5811acb0 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(378..378 src has 1..378) v4 ==== 5008+0+0 (unknown 2197200371 0 0) 0x7f5c500440b0 con 0x7f5c5811acb0 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 10 client.104210.objecter ms_dispatch 0x7f5c58000fd0 osd_map(378..378 src has 1..378) v4 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 3 client.104210.objecter handle_osd_map got epochs [378,378] > 0 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 3 client.104210.objecter handle_osd_map decoding full epoch 378 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c497fa700 20 client.104210.objecter dump_active .. 0 homeless 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.455+0000 7f5c5cbbb700 10 monclient: get_auth_request con 0x7f5c3c0387a0 auth_method 0 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c5d3bc700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6789/0 conn(0x7f5c38002840 legacy=0x7f5c38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 client.104210.objecter ms_handle_connect 0x7f5c38002840 2024-09-18T11:54:12.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 client.104210.objecter resend_mon_ops 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114872591 0 0) 0x7f5c38006590 con 0x7f5c38002840 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 monclient(hunting): my global_id is 114121 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c5811a660 con 0x7f5c38002840 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2086125185 0 0) 0x7f5c5811a660 con 0x7f5c38002840 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c38006590 con 0x7f5c38002840 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c4c002350 con 0x7f5c38002840 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 monclient: dump: 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.473 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 client.104210.objecter ms_handle_connect 0x7f5c3c0387a0 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 723317539 0 0) 0x7f5c38006590 con 0x7f5c38002840 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c3800a140 con 0x7f5c38002840 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5c4c005810 con 0x7f5c38002840 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:12.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6789/0 conn(0x7f5c38002840 legacy=0x7f5c38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.459+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6789/0 conn(0x7f5c38002840 legacy=0x7f5c38004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c5f620700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5c38007550 con 0x7f5c38006900 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c497fa700 10 client.104210.objecter ms_handle_connect 0x7f5c38006900 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c497fa700 10 client.104210.objecter resend_mon_ops 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c5d3bc700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6789/0 conn(0x7f5c38006900 legacy=0x7f5c38006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883757389 0 0) 0x7f5c38007550 con 0x7f5c38006900 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c497fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c497fa700 10 monclient(hunting): my global_id is 114124 2024-09-18T11:54:12.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.471+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c38006590 con 0x7f5c38006900 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2135021262 0 0) 0x7f5c38006590 con 0x7f5c38006900 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c38007550 con 0x7f5c38006900 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c4c0028c0 con 0x7f5c38006900 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 10 monclient: dump: 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.477 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.478 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.478 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.478 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3021149782 0 0) 0x7f5c38007550 con 0x7f5c38006900 2024-09-18T11:54:12.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5c380087b0 con 0x7f5c38006900 2024-09-18T11:54:12.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 641208449) 0x7f5c4c002b20 con 0x7f5c38006900 2024-09-18T11:54:12.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6789/0 conn(0x7f5c38006900 legacy=0x7f5c38006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c497fa700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6789/0 conn(0x7f5c38006900 legacy=0x7f5c38006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 46, 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:54:12.479 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:54:12.480 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:54:12.480 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:54:12.480 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.480 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:54:12.480 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:12.480 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:54:12.480 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:54:12.480 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:54:12.481 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:54:12.482 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:54:12.483 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:54:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:54:12.485 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:54:12.486 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:12.487 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:54:12.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:12.490 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:54:12.490 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 10 librados: watch_flush enter 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 10 librados: watch_flush exit 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 20 client.104210.objecter shutdown clearing up homeless session... 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 10 client.104210.objecter successfully canceled tick 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.129:6812/20806 conn(0x7f5c3c0387a0 legacy=0x7f5c3c03ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 10 monclient: shutdown 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 >> v1:172.21.15.112:6790/0 conn(0x7f5c5811acb0 legacy=0x7f5c58119480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5dbbd700 1 -- 172.21.15.112:0/3718338763 reap_dead start 2024-09-18T11:54:12.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5dbbd700 1 -- 172.21.15.112:0/3718338763 reap_dead start 2024-09-18T11:54:12.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 shutdown_connections 2024-09-18T11:54:12.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 >> 172.21.15.112:0/3718338763 conn(0x7f5c580680d0 msgr2=0x7f5c5806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 shutdown_connections 2024-09-18T11:54:12.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 1 -- 172.21.15.112:0/3718338763 wait complete. 2024-09-18T11:54:12.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.475+0000 7f5c5f620700 1 librados: shutdown 2024-09-18T11:54:12.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:54:12.493 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T11:54:12.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T11:54:12.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.599+0000 7f05d759e700 10 monclient: get_monmap_and_config 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.599+0000 7f05d759e700 10 monclient: build_initial_monmap 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.599+0000 7f05d759e700 10 monclient: monmap: 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:12.603332+0000 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:12.603332+0000 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:12.603 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.599+0000 7f05d759e700 1 Processor -- start 2024-09-18T11:54:12.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.599+0000 7f05d759e700 1 -- start start 2024-09-18T11:54:12.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.599+0000 7f05d759e700 10 monclient: init 2024-09-18T11:54:12.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:12.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:54:12.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 10 monclient(hunting): picked mon.noname-c con 0x7f05d80a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:12.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 10 monclient(hunting): picked mon.noname-b con 0x7f05d812edc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 10 monclient(hunting): picked mon.noname-a con 0x7f05d812f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d80981a0 con 0x7f05d812f8a0 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d806c120 con 0x7f05d80a2040 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d806eef0 con 0x7f05d812edc0 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 10 monclient(hunting): authenticate will time out at 1424.444323s 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d6d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f05d812f8a0 0x7f05d8137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54158/0 (socket says 172.21.15.112:54158) 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d5d9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f05d812edc0 0x7f05d812f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35522/0 (socket says 172.21.15.112:35522) 2024-09-18T11:54:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d6d9d700 1 -- 172.21.15.112:0/629748734 learned_addr learned my addr 172.21.15.112:0/629748734 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:12.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d6d9d700 1 -- 172.21.15.112:0/629748734 >> v1:172.21.15.112:6789/0 conn(0x7f05d812f8a0 legacy=0x7f05d8137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d5d9b700 1 -- 172.21.15.112:0/629748734 >> v1:172.21.15.112:6790/0 conn(0x7f05d812edc0 legacy=0x7f05d812f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d659c700 1 -- 172.21.15.112:0/629748734 >> v1:172.21.15.129:6789/0 conn(0x7f05d80a2040 legacy=0x7f05d8097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703175172 0 0) 0x7f05d80981a0 con 0x7f05d812f8a0 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): my global_id is 94332 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05c4003570 con 0x7f05d812f8a0 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215051259 0 0) 0x7f05d806eef0 con 0x7f05d812edc0 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): my global_id is 104213 2024-09-18T11:54:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d80981a0 con 0x7f05d812edc0 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266346745 0 0) 0x7f05d806c120 con 0x7f05d80a2040 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient(hunting): my global_id is 114127 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d806eef0 con 0x7f05d80a2040 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2326118680 0 0) 0x7f05c4003570 con 0x7f05d812f8a0 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 >> v1:172.21.15.112:6790/0 conn(0x7f05d812edc0 legacy=0x7f05d812f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 >> v1:172.21.15.129:6789/0 conn(0x7f05d80a2040 legacy=0x7f05d8097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: _finish_hunting 0 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 monclient: found mon.noname-a 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d8096dd0 con 0x7f05d812f8a0 2024-09-18T11:54:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: _finish_auth 0 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f05d0001a00 con 0x7f05d812f8a0 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: dump: 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 5 monclient: authenticate success, global_id 94332 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05d0001be0 con 0x7f05d812f8a0 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 1 -- 172.21.15.112:0/629748734 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f05d0004a90 con 0x7f05d812f8a0 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d559a700 10 monclient: dump: 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.613 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.603+0000 7f05d759e700 4 set_mon_vals no callback set 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 10 monclient: shutdown 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 -- 172.21.15.112:0/629748734 >> v1:172.21.15.112:6789/0 conn(0x7f05d812f8a0 legacy=0x7f05d8137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 -- 172.21.15.112:0/629748734 shutdown_connections 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 -- 172.21.15.112:0/629748734 >> 172.21.15.112:0/629748734 conn(0x7f05d8067da0 msgr2=0x7f05d80a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 -- 172.21.15.112:0/629748734 shutdown_connections 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 -- 172.21.15.112:0/629748734 wait complete. 2024-09-18T11:54:12.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 10 monclient: build_initial_monmap 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 10 monclient: monmap: 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:12.615 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 librados: starting msgr at 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 librados: starting objecter 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 Processor -- start 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 -- start start 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 librados: setting wanted keys 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 1 librados: calling monclient init 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 10 monclient: init 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.607+0000 7f05d759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient(hunting): picked mon.noname-b con 0x7f05d80a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient(hunting): picked mon.noname-c con 0x7f05d812edc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient(hunting): picked mon.noname-a con 0x7f05d812f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:12.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d8134270 con 0x7f05d812f8a0 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d81350f0 con 0x7f05d80a2040 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d81353d0 con 0x7f05d812edc0 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient(hunting): authenticate will time out at 1424.452129s 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d5d9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f05d812edc0 0x7f05d8133240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35538/0 (socket says 172.21.15.112:35538) 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d5d9b700 1 -- 172.21.15.112:0/3042020955 learned_addr learned my addr 172.21.15.112:0/3042020955 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d6d9d700 1 -- 172.21.15.112:0/3042020955 >> v1:172.21.15.112:6789/0 conn(0x7f05d812f8a0 legacy=0x7f05d8149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 client.?.objecter ms_handle_connect 0x7f05d812f8a0 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d5d9b700 1 -- 172.21.15.112:0/3042020955 >> v1:172.21.15.112:6790/0 conn(0x7f05d812edc0 legacy=0x7f05d8133240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 client.?.objecter ms_handle_connect 0x7f05d812edc0 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 client.?.objecter ms_handle_connect 0x7f05d80a2040 2024-09-18T11:54:12.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d659c700 1 -- 172.21.15.112:0/3042020955 >> v1:172.21.15.129:6789/0 conn(0x7f05d80a2040 legacy=0x7f05d8132b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792065319 0 0) 0x7f05d8134270 con 0x7f05d812f8a0 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): my global_id is 94335 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b8004f10 con 0x7f05d812f8a0 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731419633 0 0) 0x7f05d81353d0 con 0x7f05d812edc0 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): my global_id is 104216 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d8134270 con 0x7f05d812edc0 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199373336 0 0) 0x7f05d81350f0 con 0x7f05d80a2040 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:12.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): my global_id is 114130 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05d81353d0 con 0x7f05d80a2040 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3217610754 0 0) 0x7f05b8004f10 con 0x7f05d812f8a0 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05d81350f0 con 0x7f05d812f8a0 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f05d00019c0 con 0x7f05d812f8a0 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): dump: 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.623 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1682571968 0 0) 0x7f05d8134270 con 0x7f05d812edc0 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05b8004f10 con 0x7f05d812edc0 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f05c8002020 con 0x7f05d812edc0 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient(hunting): dump: 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.624 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3741938202 0 0) 0x7f05d81350f0 con 0x7f05d812f8a0 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 >> v1:172.21.15.112:6790/0 conn(0x7f05d812edc0 legacy=0x7f05d8133240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 >> v1:172.21.15.129:6789/0 conn(0x7f05d80a2040 legacy=0x7f05d8132b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 monclient: found mon.a 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d812e8f0 con 0x7f05d812f8a0 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 10 monclient: _finish_auth 0 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 5 monclient: authenticate success, global_id 94335 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient: _renew_subs 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d806c400 con 0x7f05d812f8a0 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 client.94335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient: _renew_subs 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d8133950 con 0x7f05d812f8a0 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.611+0000 7f05d759e700 1 librados: init done 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05d0001360 con 0x7f05d812f8a0 2024-09-18T11:54:12.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f05d0004f40 con 0x7f05d812f8a0 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05d4d99700 4 set_mon_vals no callback set 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 10 monclient: dump: 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:12.627 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f05d000f880 con 0x7f05d812f8a0 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05d759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05d759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05d759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc0050b0 con 0x7f05d812f8a0 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(378..378 src has 1..378) v4 ==== 5008+0+0 (unknown 2197200371 0 0) 0x7f05d0043d40 con 0x7f05d812f8a0 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 10 client.94335.objecter ms_dispatch 0x7f05d8000fd0 osd_map(378..378 src has 1..378) v4 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 3 client.94335.objecter handle_osd_map got epochs [378,378] > 0 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 3 client.94335.objecter handle_osd_map decoding full epoch 378 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05ce7fc700 20 client.94335.objecter dump_active .. 0 homeless 2024-09-18T11:54:12.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.615+0000 7f05d659c700 10 monclient: get_auth_request con 0x7f05b8038480 auth_method 0 2024-09-18T11:54:12.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.619+0000 7f05ce7fc700 10 client.94335.objecter ms_handle_connect 0x7f05b8038480 2024-09-18T11:54:12.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.623+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f05d00146d0 con 0x7f05d812f8a0 2024-09-18T11:54:12.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.623+0000 7f05ce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:12.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.623+0000 7f05ce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:12.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f05d759e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T11:54:12.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f05d759e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T11:54:12.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f05d759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:12.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f05bc005b70 con 0x7f05d812f8a0 2024-09-18T11:54:12.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f05ce7fc700 1 -- 172.21.15.112:0/3042020955 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f05d00144f0 con 0x7f05d812f8a0 2024-09-18T11:54:12.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f05ce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T11:54:12.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f05ce7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:12.867 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.868 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=8,logm=60} crc {auth=1007060056,config=1228700967,health=2941319762,kv=1310296871,logm=3146648811}) 2024-09-18T11:54:12.868 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.863+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=8,logm=59} crc {auth=1007060056,config=3140514009,health=2143231316,kv=1310296871,logm=2893627220}) 2024-09-18T11:54:12.868 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.868 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=8,logm=60} crc {auth=1007060056,config=1228700967,health=2941319762,kv=1310296871,logm=3146648811}) 2024-09-18T11:54:12.868 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=12,kv=7,logm=60} crc {auth=1007060056,config=3140514009,health=2941319762,kv=1757881508,logm=2244869247}) 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 10 librados: watch_flush enter 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 10 librados: watch_flush exit 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 20 client.94335.objecter shutdown clearing up homeless session... 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 10 client.94335.objecter successfully canceled tick 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 >> v1:172.21.15.129:6812/20806 conn(0x7f05b8038480 legacy=0x7f05b803a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 10 monclient: shutdown 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 >> v1:172.21.15.112:6789/0 conn(0x7f05d812f8a0 legacy=0x7f05d8149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:12.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 shutdown_connections 2024-09-18T11:54:12.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 >> 172.21.15.112:0/3042020955 conn(0x7f05d8067da0 msgr2=0x7f05d8098630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:12.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 shutdown_connections 2024-09-18T11:54:12.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 1 -- 172.21.15.112:0/3042020955 wait complete. 2024-09-18T11:54:12.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f05d759e700 1 librados: shutdown 2024-09-18T11:54:12.871 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.871 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2657274509}) 2024-09-18T11:54:12.871 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011848288}) 2024-09-18T11:54:12.871 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.871 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2657274509}) 2024-09-18T11:54:12.871 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.867+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1875059941}) 2024-09-18T11:54:12.873 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.873 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610716631}) 2024-09-18T11:54:12.873 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2574780802}) 2024-09-18T11:54:12.873 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.873 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610716631}) 2024-09-18T11:54:12.873 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3192061083}) 2024-09-18T11:54:12.874 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.874 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706387968}) 2024-09-18T11:54:12.874 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1848217727}) 2024-09-18T11:54:12.874 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706387968}) 2024-09-18T11:54:12.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1666327205}) 2024-09-18T11:54:12.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2659707360}) 2024-09-18T11:54:12.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889684957}) 2024-09-18T11:54:12.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.876 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2659707360}) 2024-09-18T11:54:12.876 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.871+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1642175107}) 2024-09-18T11:54:12.877 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.877 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879555142}) 2024-09-18T11:54:12.877 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515710850}) 2024-09-18T11:54:12.877 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.877 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879555142}) 2024-09-18T11:54:12.877 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897889709}) 2024-09-18T11:54:12.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223873065}) 2024-09-18T11:54:12.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=127167071}) 2024-09-18T11:54:12.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223873065}) 2024-09-18T11:54:12.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3602656180}) 2024-09-18T11:54:12.879 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.879 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1257240170}) 2024-09-18T11:54:12.879 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3022561599}) 2024-09-18T11:54:12.879 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.879 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1257240170}) 2024-09-18T11:54:12.879 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.875+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=870715652}) 2024-09-18T11:54:12.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T11:54:12.880 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.881 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408920576}) 2024-09-18T11:54:12.881 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3639586401}) 2024-09-18T11:54:12.881 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.881 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408920576}) 2024-09-18T11:54:12.881 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690041285}) 2024-09-18T11:54:12.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182556866}) 2024-09-18T11:54:12.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989758848}) 2024-09-18T11:54:12.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182556866}) 2024-09-18T11:54:12.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720917284}) 2024-09-18T11:54:12.883 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.883 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919656917}) 2024-09-18T11:54:12.883 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355589769}) 2024-09-18T11:54:12.883 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.883 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919656917}) 2024-09-18T11:54:12.883 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.879+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2327934714}) 2024-09-18T11:54:12.884 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.884 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346790542}) 2024-09-18T11:54:12.884 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1152120470}) 2024-09-18T11:54:12.884 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.884 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346790542}) 2024-09-18T11:54:12.884 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462532656}) 2024-09-18T11:54:12.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167094362}) 2024-09-18T11:54:12.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1329721534}) 2024-09-18T11:54:12.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.886 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167094362}) 2024-09-18T11:54:12.886 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844969127}) 2024-09-18T11:54:12.887 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.887 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219055733}) 2024-09-18T11:54:12.887 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512580042}) 2024-09-18T11:54:12.887 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.887 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219055733}) 2024-09-18T11:54:12.887 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.883+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279299661}) 2024-09-18T11:54:12.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033084509}) 2024-09-18T11:54:12.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=177438107}) 2024-09-18T11:54:12.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033084509}) 2024-09-18T11:54:12.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1262169487}) 2024-09-18T11:54:12.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176284920}) 2024-09-18T11:54:12.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280017435}) 2024-09-18T11:54:12.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176284920}) 2024-09-18T11:54:12.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=780457182}) 2024-09-18T11:54:12.890 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.890 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481619115}) 2024-09-18T11:54:12.890 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3517487484}) 2024-09-18T11:54:12.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481619115}) 2024-09-18T11:54:12.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.887+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1758221999}) 2024-09-18T11:54:12.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123271655}) 2024-09-18T11:54:12.892 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842743720}) 2024-09-18T11:54:12.892 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.892 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123271655}) 2024-09-18T11:54:12.892 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040811917}) 2024-09-18T11:54:12.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087191743}) 2024-09-18T11:54:12.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221968123}) 2024-09-18T11:54:12.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087191743}) 2024-09-18T11:54:12.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4116323331}) 2024-09-18T11:54:12.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859579436}) 2024-09-18T11:54:12.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770832195}) 2024-09-18T11:54:12.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859579436}) 2024-09-18T11:54:12.895 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1514018702}) 2024-09-18T11:54:12.895 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.895 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4154809107}) 2024-09-18T11:54:12.895 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=658406562}) 2024-09-18T11:54:12.895 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.896 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4154809107}) 2024-09-18T11:54:12.896 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.891+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464470555}) 2024-09-18T11:54:12.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782259332}) 2024-09-18T11:54:12.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149978350}) 2024-09-18T11:54:12.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782259332}) 2024-09-18T11:54:12.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429089202}) 2024-09-18T11:54:12.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138597183}) 2024-09-18T11:54:12.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1449469033}) 2024-09-18T11:54:12.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138597183}) 2024-09-18T11:54:12.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021487997}) 2024-09-18T11:54:12.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996932273}) 2024-09-18T11:54:12.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4265616572}) 2024-09-18T11:54:12.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996932273}) 2024-09-18T11:54:12.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.895+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383801116}) 2024-09-18T11:54:12.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718019075}) 2024-09-18T11:54:12.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2790261540}) 2024-09-18T11:54:12.901 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.901 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718019075}) 2024-09-18T11:54:12.901 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258180708}) 2024-09-18T11:54:12.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406437308}) 2024-09-18T11:54:12.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1684129343}) 2024-09-18T11:54:12.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406437308}) 2024-09-18T11:54:12.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1736717582}) 2024-09-18T11:54:12.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143698788}) 2024-09-18T11:54:12.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3697404870}) 2024-09-18T11:54:12.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143698788}) 2024-09-18T11:54:12.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.899+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2308601808}) 2024-09-18T11:54:12.904 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.903+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.904 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.903+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415498905}) 2024-09-18T11:54:12.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.903+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1529444982}) 2024-09-18T11:54:12.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.903+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.903+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415498905}) 2024-09-18T11:54:12.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.903+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2878788171}) 2024-09-18T11:54:12.914 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.911+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.914 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.911+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2082236661}) 2024-09-18T11:54:12.915 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.911+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4278237229}) 2024-09-18T11:54:12.915 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.911+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.915 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.911+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2082236661}) 2024-09-18T11:54:12.915 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.911+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2836856838}) 2024-09-18T11:54:12.916 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.916 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683141722}) 2024-09-18T11:54:12.916 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2174195888}) 2024-09-18T11:54:12.916 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.917 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683141722}) 2024-09-18T11:54:12.917 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1917424403}) 2024-09-18T11:54:12.918 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.919 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028382285}) 2024-09-18T11:54:12.919 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1844975887}) 2024-09-18T11:54:12.919 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.919 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028382285}) 2024-09-18T11:54:12.919 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.915+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416134001}) 2024-09-18T11:54:12.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133340880}) 2024-09-18T11:54:12.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3932701821}) 2024-09-18T11:54:12.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133340880}) 2024-09-18T11:54:12.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110580412}) 2024-09-18T11:54:12.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496426006}) 2024-09-18T11:54:12.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=289150201}) 2024-09-18T11:54:12.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496426006}) 2024-09-18T11:54:12.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.919+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367863703}) 2024-09-18T11:54:12.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623439849}) 2024-09-18T11:54:12.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156817753}) 2024-09-18T11:54:12.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623439849}) 2024-09-18T11:54:12.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417659194}) 2024-09-18T11:54:12.926 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.926 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014086803}) 2024-09-18T11:54:12.926 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3020044780}) 2024-09-18T11:54:12.926 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.926 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014086803}) 2024-09-18T11:54:12.926 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407121110}) 2024-09-18T11:54:12.928 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.923+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.928 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445142064}) 2024-09-18T11:54:12.928 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813148500}) 2024-09-18T11:54:12.928 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.928 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1445142064}) 2024-09-18T11:54:12.928 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=172887908}) 2024-09-18T11:54:12.929 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.929 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226373253}) 2024-09-18T11:54:12.929 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848236137}) 2024-09-18T11:54:12.929 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.929 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226373253}) 2024-09-18T11:54:12.929 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=528432090}) 2024-09-18T11:54:12.930 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.931 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232073192}) 2024-09-18T11:54:12.931 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526123721}) 2024-09-18T11:54:12.931 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.931 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232073192}) 2024-09-18T11:54:12.931 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.927+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857936863}) 2024-09-18T11:54:12.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330762248}) 2024-09-18T11:54:12.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=411401624}) 2024-09-18T11:54:12.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330762248}) 2024-09-18T11:54:12.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680627485}) 2024-09-18T11:54:12.933 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.933 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657980819}) 2024-09-18T11:54:12.933 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=455219766,mdsmap=2826658659}) 2024-09-18T11:54:12.933 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.934 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657980819}) 2024-09-18T11:54:12.934 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=844022679}) 2024-09-18T11:54:12.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=22} crc {logm=3578722681,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1655026099,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=3594653274,osd_snap=4024326838,osdmap=4275828497}) 2024-09-18T11:54:12.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=8,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=36,osdmap=23} crc {logm=2030827740,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgrstat=1724086723,mon_config_key=3035149395,monmap=1725997317,osd_metadata=1765517589,osd_pg_creating=3594653274,osd_snap=3941203679,osdmap=1441137185}) 2024-09-18T11:54:12.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=22} crc {logm=3578722681,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1655026099,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=3594653274,osd_snap=4024326838,osdmap=4275828497}) 2024-09-18T11:54:12.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.931+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=7,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=23} crc {logm=646876459,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1155547317,mon_config_key=868353578,monmap=3140514009,osd_metadata=871764258,osd_pg_creating=3594653274,osd_snap=2208427651,osdmap=417986559}) 2024-09-18T11:54:12.937 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.937 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1535208575}) 2024-09-18T11:54:12.937 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=211228041}) 2024-09-18T11:54:12.937 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1535208575}) 2024-09-18T11:54:12.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1443916758}) 2024-09-18T11:54:12.939 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.939 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2740970783}) 2024-09-18T11:54:12.939 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3732794485}) 2024-09-18T11:54:12.939 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.939 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2740970783}) 2024-09-18T11:54:12.939 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.935+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=664365691}) 2024-09-18T11:54:12.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1086677442}) 2024-09-18T11:54:12.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2441745577}) 2024-09-18T11:54:12.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1086677442}) 2024-09-18T11:54:12.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2762188834}) 2024-09-18T11:54:12.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3176313390}) 2024-09-18T11:54:12.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1460458107}) 2024-09-18T11:54:12.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3176313390}) 2024-09-18T11:54:12.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.939+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1346585911}) 2024-09-18T11:54:12.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.943+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.943+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1855549924}) 2024-09-18T11:54:12.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.943+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3472992537}) 2024-09-18T11:54:12.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.943+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.943+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1855549924}) 2024-09-18T11:54:12.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.943+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=204121340}) 2024-09-18T11:54:12.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.947+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.947+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=34} crc {osdmap=1379723258}) 2024-09-18T11:54:12.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.947+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=2950999008}) 2024-09-18T11:54:12.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.947+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:54:12.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.947+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=34} crc {osdmap=1379723258}) 2024-09-18T11:54:12.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:54:12.947+0000 7f0092408700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=34} crc {osdmap=2304378047}) 2024-09-18T11:54:13.881 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:54:13.881 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:54:13.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient: get_monmap_and_config 2024-09-18T11:54:13.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient: build_initial_monmap 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient: monmap: 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:13.990086+0000 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:13.990086+0000 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:13.990 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:13.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 1 Processor -- start 2024-09-18T11:54:13.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 1 -- start start 2024-09-18T11:54:13.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient: init 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient(hunting): picked mon.noname-b con 0x7f6f280a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient(hunting): picked mon.noname-c con 0x7f6f2806c280 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient(hunting): picked mon.noname-a con 0x7f6f2806c670 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2806e950 con 0x7f6f2806c670 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f280a2270 con 0x7f6f2806c280 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2806d130 con 0x7f6f280a1ea0 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2759e700 10 monclient(hunting): authenticate will time out at 1425.831183s 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f6f280a1ea0 0x7f6f2806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35550/0 (socket says 172.21.15.112:35550) 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f26d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6f2806c670 0x7f6f281388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54182/0 (socket says 172.21.15.112:54182) 2024-09-18T11:54:13.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.987+0000 7f6f2659c700 1 -- 172.21.15.112:0/1179910903 learned_addr learned my addr 172.21.15.112:0/1179910903 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:13.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f26d9d700 1 -- 172.21.15.112:0/1179910903 >> v1:172.21.15.112:6789/0 conn(0x7f6f2806c670 legacy=0x7f6f281388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:13.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2659c700 1 -- 172.21.15.112:0/1179910903 >> v1:172.21.15.112:6790/0 conn(0x7f6f280a1ea0 legacy=0x7f6f2806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f25d9b700 1 -- 172.21.15.112:0/1179910903 >> v1:172.21.15.129:6789/0 conn(0x7f6f2806c280 legacy=0x7f6f28135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200467771 0 0) 0x7f6f2806d130 con 0x7f6f280a1ea0 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): my global_id is 104219 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f10003570 con 0x7f6f280a1ea0 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598336629 0 0) 0x7f6f2806e950 con 0x7f6f2806c670 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): my global_id is 94338 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f2806d130 con 0x7f6f2806c670 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252525462 0 0) 0x7f6f280a2270 con 0x7f6f2806c280 2024-09-18T11:54:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient(hunting): my global_id is 114133 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f2806e950 con 0x7f6f2806c280 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1967711698 0 0) 0x7f6f10003570 con 0x7f6f280a1ea0 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 >> v1:172.21.15.129:6789/0 conn(0x7f6f2806c280 legacy=0x7f6f28135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 >> v1:172.21.15.112:6789/0 conn(0x7f6f2806c670 legacy=0x7f6f281388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: _finish_hunting 0 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 monclient: found mon.noname-b 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2809e370 con 0x7f6f280a1ea0 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: _finish_auth 0 2024-09-18T11:54:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6f18001940 con 0x7f6f280a1ea0 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 5 monclient: authenticate success, global_id 104219 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: dump: 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:13.997 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f18001bd0 con 0x7f6f280a1ea0 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 4 set_mon_vals no callback set 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 1 -- 172.21.15.112:0/1179910903 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6f18004a90 con 0x7f6f280a1ea0 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2559a700 10 monclient: dump: 2024-09-18T11:54:13.998 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:13.999 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 10 monclient: shutdown 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 1 -- 172.21.15.112:0/1179910903 >> v1:172.21.15.112:6790/0 conn(0x7f6f280a1ea0 legacy=0x7f6f2806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 1 -- 172.21.15.112:0/1179910903 shutdown_connections 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.991+0000 7f6f2759e700 1 -- 172.21.15.112:0/1179910903 >> 172.21.15.112:0/1179910903 conn(0x7f6f28067da0 msgr2=0x7f6f280a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 -- 172.21.15.112:0/1179910903 shutdown_connections 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 -- 172.21.15.112:0/1179910903 wait complete. 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient: monmap: 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:14.003 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 librados: starting msgr at 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 librados: starting objecter 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 Processor -- start 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 -- start start 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 librados: setting wanted keys 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 librados: calling monclient init 2024-09-18T11:54:14.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient: init 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient(hunting): picked mon.noname-a con 0x7f6f2806c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient(hunting): picked mon.noname-b con 0x7f6f2806c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient(hunting): picked mon.noname-c con 0x7f6f280a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f28132d90 con 0x7f6f2806c280 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f281421b0 con 0x7f6f2806c670 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f28121410 con 0x7f6f280a1ea0 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2759e700 10 monclient(hunting): authenticate will time out at 1425.838842s 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6f2806c280 0x7f6f28131ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54194/0 (socket says 172.21.15.112:54194) 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.995+0000 7f6f2659c700 1 -- 172.21.15.112:0/3101552851 learned_addr learned my addr 172.21.15.112:0/3101552851 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2659c700 1 -- 172.21.15.112:0/3101552851 >> v1:172.21.15.112:6789/0 conn(0x7f6f2806c280 legacy=0x7f6f28131ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 client.?.objecter ms_handle_connect 0x7f6f2806c280 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 client.?.objecter ms_handle_connect 0x7f6f280a1ea0 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f26d9d700 1 -- 172.21.15.112:0/3101552851 >> v1:172.21.15.112:6790/0 conn(0x7f6f280a1ea0 legacy=0x7f6f28149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159946852 0 0) 0x7f6f28132d90 con 0x7f6f2806c280 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): my global_id is 94341 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f08004f10 con 0x7f6f2806c280 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f25d9b700 1 -- 172.21.15.112:0/3101552851 >> v1:172.21.15.129:6789/0 conn(0x7f6f2806c670 legacy=0x7f6f28146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 client.?.objecter ms_handle_connect 0x7f6f2806c670 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950852554 0 0) 0x7f6f28121410 con 0x7f6f280a1ea0 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): my global_id is 104222 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f28132d90 con 0x7f6f280a1ea0 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415179967 0 0) 0x7f6f281421b0 con 0x7f6f2806c670 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): my global_id is 114136 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f28121410 con 0x7f6f2806c670 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393608946 0 0) 0x7f6f08004f10 con 0x7f6f2806c280 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f281421b0 con 0x7f6f2806c280 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6f18001720 con 0x7f6f2806c280 2024-09-18T11:54:14.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): dump: 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:14.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2641178502 0 0) 0x7f6f28132d90 con 0x7f6f280a1ea0 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f08004f10 con 0x7f6f280a1ea0 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6f20001cf0 con 0x7f6f280a1ea0 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient(hunting): dump: 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.009 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2298406782 0 0) 0x7f6f28121410 con 0x7f6f2806c670 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f28132d90 con 0x7f6f2806c670 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 899253747 0 0) 0x7f6f281421b0 con 0x7f6f2806c280 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 >> v1:172.21.15.112:6790/0 conn(0x7f6f280a1ea0 legacy=0x7f6f28149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 >> v1:172.21.15.129:6789/0 conn(0x7f6f2806c670 legacy=0x7f6f28146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 monclient: found mon.a 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f281219d0 con 0x7f6f2806c280 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 5 monclient: authenticate success, global_id 94341 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 10 monclient: _renew_subs 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f28132700 con 0x7f6f2806c280 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 10 client.94341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 10 monclient: _renew_subs 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f281203e0 con 0x7f6f2806c280 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 1 librados: init done 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f18004f40 con 0x7f6f2806c280 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6f18005250 con 0x7f6f2806c280 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f24d99700 4 set_mon_vals no callback set 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 10 monclient: dump: 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.012 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.013 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.013 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f6f1800fad0 con 0x7f6f2806c280 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:13.999+0000 7f6f2759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f2759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f2759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c0050b0 con 0x7f6f2806c280 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(380..380 src has 1..380) v4 ==== 5393+0+0 (unknown 4241097072 0 0) 0x7f6f18044050 con 0x7f6f2806c280 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f167fc700 10 client.94341.objecter ms_dispatch 0x7f6f28000fd0 osd_map(380..380 src has 1..380) v4 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f167fc700 3 client.94341.objecter handle_osd_map got epochs [380,380] > 0 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f167fc700 3 client.94341.objecter handle_osd_map decoding full epoch 380 2024-09-18T11:54:14.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f167fc700 20 client.94341.objecter dump_active .. 0 homeless 2024-09-18T11:54:14.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f25d9b700 10 monclient: get_auth_request con 0x7f6f08038600 auth_method 0 2024-09-18T11:54:14.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.003+0000 7f6f167fc700 10 client.94341.objecter ms_handle_connect 0x7f6f08038600 2024-09-18T11:54:14.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.007+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f6f18014c80 con 0x7f6f2806c280 2024-09-18T11:54:14.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.007+0000 7f6f167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.007+0000 7f6f167fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.227+0000 7f6f2759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:54:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.227+0000 7f6f2759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:54:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.227+0000 7f6f2759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.227+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6f0c0058e0 con 0x7f6f2806c280 2024-09-18T11:54:14.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.227+0000 7f6f167fc700 1 -- 172.21.15.112:0/3101552851 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1851202094) 0x7f6f18014680 con 0x7f6f2806c280 2024-09-18T11:54:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.227+0000 7f6f167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:54:14.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.227+0000 7f6f167fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:14.231 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:54:14.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 10 librados: watch_flush enter 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 10 librados: watch_flush exit 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 20 client.94341.objecter shutdown clearing up homeless session... 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 10 client.94341.objecter successfully canceled tick 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 >> v1:172.21.15.129:6812/20806 conn(0x7f6f08038600 legacy=0x7f6f0803aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 10 monclient: shutdown 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 >> v1:172.21.15.112:6789/0 conn(0x7f6f2806c280 legacy=0x7f6f28131ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 shutdown_connections 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 >> 172.21.15.112:0/3101552851 conn(0x7f6f28067da0 msgr2=0x7f6f2809fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 shutdown_connections 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 1 -- 172.21.15.112:0/3101552851 wait complete. 2024-09-18T11:54:14.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.231+0000 7f6f2759e700 1 librados: shutdown 2024-09-18T11:54:14.246 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:54:14.246 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T11:54:14.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:54:14.246 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:54:14.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.347+0000 7fc2519d6700 10 monclient: get_monmap_and_config 2024-09-18T11:54:14.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.347+0000 7fc2519d6700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.347+0000 7fc2519d6700 10 monclient: monmap: 2024-09-18T11:54:14.351 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.351 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.351 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:14.351876+0000 2024-09-18T11:54:14.352 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:14.351876+0000 2024-09-18T11:54:14.352 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.352 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.352 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.352 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:14.352 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:14.352 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 1 Processor -- start 2024-09-18T11:54:14.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 1 -- start start 2024-09-18T11:54:14.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 10 monclient: init 2024-09-18T11:54:14.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:54:14.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 10 monclient(hunting): picked mon.noname-b con 0x7fc24c067f10 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 10 monclient(hunting): picked mon.noname-a con 0x7fc24c06eac0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 10 monclient(hunting): picked mon.noname-c con 0x7fc24c0689f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c096f90 con 0x7fc24c06eac0 2024-09-18T11:54:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c097270 con 0x7fc24c0689f0 2024-09-18T11:54:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c098230 con 0x7fc24c067f10 2024-09-18T11:54:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2519d6700 10 monclient(hunting): authenticate will time out at 1426.193275s 2024-09-18T11:54:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2509d4700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc24c067f10 0x7fc24c0682e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35564/0 (socket says 172.21.15.112:35564) 2024-09-18T11:54:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2509d4700 1 -- 172.21.15.112:0/642899478 learned_addr learned my addr 172.21.15.112:0/642899478 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc2509d4700 1 -- 172.21.15.112:0/642899478 >> v1:172.21.15.112:6790/0 conn(0x7fc24c067f10 legacy=0x7fc24c0682e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.351+0000 7fc24bfff700 1 -- 172.21.15.112:0/642899478 >> v1:172.21.15.112:6789/0 conn(0x7fc24c06eac0 legacy=0x7fc24c1354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175520370 0 0) 0x7fc24c098230 con 0x7fc24c067f10 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient(hunting): my global_id is 104225 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2511d5700 1 -- 172.21.15.112:0/642899478 >> v1:172.21.15.129:6789/0 conn(0x7fc24c0689f0 legacy=0x7fc24c138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc22c003570 con 0x7fc24c067f10 2024-09-18T11:54:14.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242244498 0 0) 0x7fc24c096f90 con 0x7fc24c06eac0 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient(hunting): my global_id is 94344 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc24c098230 con 0x7fc24c06eac0 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608552280 0 0) 0x7fc22c003570 con 0x7fc24c067f10 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 >> v1:172.21.15.129:6789/0 conn(0x7fc24c0689f0 legacy=0x7fc24c138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 >> v1:172.21.15.112:6789/0 conn(0x7fc24c06eac0 legacy=0x7fc24c1354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 monclient: found mon.noname-b 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c0a3090 con 0x7fc24c067f10 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 5 monclient: authenticate success, global_id 104225 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc234001940 con 0x7fc24c067f10 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: dump: 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.359 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc234001bd0 con 0x7fc24c067f10 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 1 -- 172.21.15.112:0/642899478 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc234004a90 con 0x7fc24c067f10 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 4 set_mon_vals no callback set 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc24b7fe700 10 monclient: dump: 2024-09-18T11:54:14.360 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient: shutdown 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- 172.21.15.112:0/642899478 >> v1:172.21.15.112:6790/0 conn(0x7fc24c067f10 legacy=0x7fc24c0682e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- 172.21.15.112:0/642899478 shutdown_connections 2024-09-18T11:54:14.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- 172.21.15.112:0/642899478 >> 172.21.15.112:0/642899478 conn(0x7fc24c069e20 msgr2=0x7fc24c0a2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- 172.21.15.112:0/642899478 shutdown_connections 2024-09-18T11:54:14.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- 172.21.15.112:0/642899478 wait complete. 2024-09-18T11:54:14.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient: monmap: 2024-09-18T11:54:14.368 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 librados: starting msgr at 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 librados: starting objecter 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 Processor -- start 2024-09-18T11:54:14.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- start start 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 librados: setting wanted keys 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 librados: calling monclient init 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient: init 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient(hunting): picked mon.noname-c con 0x7fc24c067f10 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient(hunting): picked mon.noname-b con 0x7fc24c0689f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient(hunting): picked mon.noname-a con 0x7fc24c06eac0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c098980 con 0x7fc24c06eac0 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c121a10 con 0x7fc24c0689f0 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c06efc0 con 0x7fc24c067f10 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2519d6700 10 monclient(hunting): authenticate will time out at 1426.198947s 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2509d4700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc24c067f10 0x7fc24c141720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35566/0 (socket says 172.21.15.112:35566) 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.355+0000 7fc2509d4700 1 -- 172.21.15.112:0/2499562480 learned_addr learned my addr 172.21.15.112:0/2499562480 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2509d4700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6790/0 conn(0x7fc24c067f10 legacy=0x7fc24c141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 client.?.objecter ms_handle_connect 0x7fc24c067f10 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 client.?.objecter ms_handle_connect 0x7fc24c06eac0 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2511d5700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6789/0 conn(0x7fc24c06eac0 legacy=0x7fc24c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816714054 0 0) 0x7fc24c06efc0 con 0x7fc24c067f10 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): my global_id is 104228 2024-09-18T11:54:14.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc230004f10 con 0x7fc24c067f10 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 client.?.objecter ms_handle_connect 0x7fc24c0689f0 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc24bfff700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.129:6789/0 conn(0x7fc24c0689f0 legacy=0x7fc24c1385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372413493 0 0) 0x7fc24c098980 con 0x7fc24c06eac0 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): my global_id is 94347 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc24c06efc0 con 0x7fc24c06eac0 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1958444799 0 0) 0x7fc230004f10 con 0x7fc24c067f10 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc24c098980 con 0x7fc24c067f10 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc2340049d0 con 0x7fc24c067f10 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:54:14.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): dump: 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837002349 0 0) 0x7fc24c121a10 con 0x7fc24c0689f0 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): my global_id is 114142 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc230004f10 con 0x7fc24c0689f0 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1872348605 0 0) 0x7fc24c06efc0 con 0x7fc24c06eac0 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc24c121a10 con 0x7fc24c06eac0 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc23c001bc0 con 0x7fc24c06eac0 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient(hunting): dump: 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.374 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3907402255 0 0) 0x7fc24c098980 con 0x7fc24c067f10 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.129:6789/0 conn(0x7fc24c0689f0 legacy=0x7fc24c1385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6789/0 conn(0x7fc24c06eac0 legacy=0x7fc24c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 monclient: found mon.c 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:14.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c1425b0 con 0x7fc24c067f10 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 5 monclient: authenticate success, global_id 104228 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 10 monclient: _renew_subs 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24c06d400 con 0x7fc24c067f10 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 10 client.104228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 10 monclient: _renew_subs 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24c138cd0 con 0x7fc24c067f10 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 1 librados: init done 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc234001580 con 0x7fc24c067f10 2024-09-18T11:54:14.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc2340051e0 con 0x7fc24c067f10 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc24affd700 4 set_mon_vals no callback set 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc248ff9700 10 monclient: dump: 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.377 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.359+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc224006590 con 0x7fc224002840 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fc23400fae0 con 0x7fc24c067f10 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(380..380 src has 1..380) v4 ==== 5393+0+0 (unknown 4241097072 0 0) 0x7fc2340441a0 con 0x7fc24c067f10 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 client.104228.objecter ms_dispatch 0x7fc24c000fd0 osd_map(380..380 src has 1..380) v4 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 3 client.104228.objecter handle_osd_map got epochs [380,380] > 0 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 3 client.104228.objecter handle_osd_map decoding full epoch 380 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 20 client.104228.objecter dump_active .. 0 homeless 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc2511d5700 10 monclient: get_auth_request con 0x7fc230038430 auth_method 0 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 client.104228.objecter ms_handle_connect 0x7fc224002840 2024-09-18T11:54:14.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 client.104228.objecter resend_mon_ops 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc24bfff700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6789/0 conn(0x7fc224002840 legacy=0x7fc224004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617609090 0 0) 0x7fc224006590 con 0x7fc224002840 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 monclient(hunting): my global_id is 94350 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc24c098980 con 0x7fc224002840 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 104873298 0 0) 0x7fc24c098980 con 0x7fc224002840 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc224006590 con 0x7fc224002840 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc244001e80 con 0x7fc224002840 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 monclient: dump: 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.379 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4148582791 0 0) 0x7fc224006590 con 0x7fc224002840 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc22400a140 con 0x7fc224002840 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fc244005b90 con 0x7fc224002840 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:14.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6789/0 conn(0x7fc224002840 legacy=0x7fc224004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6789/0 conn(0x7fc224002840 legacy=0x7fc224004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:14.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.363+0000 7fc248ff9700 10 client.104228.objecter ms_handle_connect 0x7fc230038430 2024-09-18T11:54:14.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.375+0000 7fc2519d6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:54:14.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.375+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc224009670 con 0x7fc2240090e0 2024-09-18T11:54:14.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc24bfff700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6789/0 conn(0x7fc2240090e0 legacy=0x7fc224008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 client.104228.objecter ms_handle_connect 0x7fc2240090e0 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 client.104228.objecter resend_mon_ops 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671871471 0 0) 0x7fc224009670 con 0x7fc2240090e0 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 monclient(hunting): my global_id is 94353 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc224006590 con 0x7fc2240090e0 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3404100697 0 0) 0x7fc224006590 con 0x7fc2240090e0 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc224009670 con 0x7fc2240090e0 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc244002710 con 0x7fc2240090e0 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 monclient: dump: 2024-09-18T11:54:14.383 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 968309070 0 0) 0x7fc224009670 con 0x7fc2240090e0 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc224009930 con 0x7fc2240090e0 2024-09-18T11:54:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4173 (unknown 0 0 795400048) 0x7fc244002870 con 0x7fc2240090e0 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6789/0 conn(0x7fc2240090e0 legacy=0x7fc224008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc248ff9700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6789/0 conn(0x7fc2240090e0 legacy=0x7fc224008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 46, 2024-09-18T11:54:14.385 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.386 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.387 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:14.388 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.389 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:14.390 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.391 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.392 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:54:14.394 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:14.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.396 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:54:14.397 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 10 librados: watch_flush enter 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 10 librados: watch_flush exit 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 20 client.104228.objecter shutdown clearing up homeless session... 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 10 client.104228.objecter successfully canceled tick 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.129:6812/20806 conn(0x7fc230038430 legacy=0x7fc23003a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 10 monclient: shutdown 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 >> v1:172.21.15.112:6790/0 conn(0x7fc24c067f10 legacy=0x7fc24c141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2511d5700 1 -- 172.21.15.112:0/2499562480 reap_dead start 2024-09-18T11:54:14.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 shutdown_connections 2024-09-18T11:54:14.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 >> 172.21.15.112:0/2499562480 conn(0x7fc24c069e20 msgr2=0x7fc24c139540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 shutdown_connections 2024-09-18T11:54:14.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 1 -- 172.21.15.112:0/2499562480 wait complete. 2024-09-18T11:54:14.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.379+0000 7fc2519d6700 1 librados: shutdown 2024-09-18T11:54:14.401 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:54:14.401 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:54:14.401 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:54:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient: get_monmap_and_config 2024-09-18T11:54:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient: monmap: 2024-09-18T11:54:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:14.510649+0000 2024-09-18T11:54:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:14.510649+0000 2024-09-18T11:54:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:14.511 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 1 Processor -- start 2024-09-18T11:54:14.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 1 -- start start 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient: init 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient(hunting): picked mon.noname-b con 0x7fa8c806a830 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient(hunting): picked mon.noname-c con 0x7fa8c809bcc0 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 10 monclient(hunting): picked mon.noname-a con 0x7fa8c8135300 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c809fe30 con 0x7fa8c8135300 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.507+0000 7fa8cf149700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c809faa0 con 0x7fa8c809bcc0 2024-09-18T11:54:14.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cf149700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c806c6f0 con 0x7fa8c806a830 2024-09-18T11:54:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cf149700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cf149700 10 monclient(hunting): authenticate will time out at 1426.352043s 2024-09-18T11:54:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8ccee5700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa8c806a830 0x7fa8c806ac00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35580/0 (socket says 172.21.15.112:35580) 2024-09-18T11:54:14.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8ccee5700 1 -- 172.21.15.112:0/1293323849 learned_addr learned my addr 172.21.15.112:0/1293323849 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8ccee5700 1 -- 172.21.15.112:0/1293323849 >> v1:172.21.15.112:6790/0 conn(0x7fa8c806a830 legacy=0x7fa8c806ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cd6e6700 1 -- 172.21.15.112:0/1293323849 >> v1:172.21.15.112:6789/0 conn(0x7fa8c8135300 legacy=0x7fa8c81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c7fff700 1 -- 172.21.15.112:0/1293323849 >> v1:172.21.15.129:6789/0 conn(0x7fa8c809bcc0 legacy=0x7fa8c806b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792347976 0 0) 0x7fa8c809fe30 con 0x7fa8c8135300 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): my global_id is 94356 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a8003570 con 0x7fa8c8135300 2024-09-18T11:54:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252277096 0 0) 0x7fa8c806c6f0 con 0x7fa8c806a830 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): my global_id is 104231 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c809fe30 con 0x7fa8c806a830 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299266550 0 0) 0x7fa8c809faa0 con 0x7fa8c809bcc0 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient(hunting): my global_id is 114145 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c806c6f0 con 0x7fa8c809bcc0 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789352404 0 0) 0x7fa8a8003570 con 0x7fa8c8135300 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 >> v1:172.21.15.112:6790/0 conn(0x7fa8c806a830 legacy=0x7fa8c806ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 >> v1:172.21.15.129:6789/0 conn(0x7fa8c809bcc0 legacy=0x7fa8c806b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 monclient: found mon.noname-a 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c806d980 con 0x7fa8c8135300 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cf149700 5 monclient: authenticate success, global_id 94356 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8c0001d10 con 0x7fa8c8135300 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cf149700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: dump: 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cf149700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8c0001fa0 con 0x7fa8c8135300 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 1 -- 172.21.15.112:0/1293323849 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8c0002420 con 0x7fa8c8135300 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cf149700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:14.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8cf149700 4 set_mon_vals no callback set 2024-09-18T11:54:14.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.511+0000 7fa8c77fe700 10 monclient: dump: 2024-09-18T11:54:14.521 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.521 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.521 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.522 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.522 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.522 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.522 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.522 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.522 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.522 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient: shutdown 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- 172.21.15.112:0/1293323849 >> v1:172.21.15.112:6789/0 conn(0x7fa8c8135300 legacy=0x7fa8c81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- 172.21.15.112:0/1293323849 shutdown_connections 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- 172.21.15.112:0/1293323849 >> 172.21.15.112:0/1293323849 conn(0x7fa8c80680d0 msgr2=0x7fa8c80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- 172.21.15.112:0/1293323849 shutdown_connections 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- 172.21.15.112:0/1293323849 wait complete. 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient: monmap: 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.523 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.524 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.524 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:14.524 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:14.524 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 librados: starting msgr at 2024-09-18T11:54:14.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 librados: starting objecter 2024-09-18T11:54:14.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 Processor -- start 2024-09-18T11:54:14.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- start start 2024-09-18T11:54:14.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 librados: setting wanted keys 2024-09-18T11:54:14.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 librados: calling monclient init 2024-09-18T11:54:14.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient: init 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient(hunting): picked mon.noname-c con 0x7fa8c806a830 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient(hunting): picked mon.noname-a con 0x7fa8c809bcc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient(hunting): picked mon.noname-b con 0x7fa8c8135300 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c806efb0 con 0x7fa8c809bcc0 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c8138b40 con 0x7fa8c8135300 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c811f310 con 0x7fa8c806a830 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8c7fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa8c809bcc0 0x7fa8c811ca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54262/0 (socket says 172.21.15.112:54262) 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8c7fff700 1 -- 172.21.15.112:0/3435450341 learned_addr learned my addr 172.21.15.112:0/3435450341 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.515+0000 7fa8cf149700 10 monclient(hunting): authenticate will time out at 1426.359271s 2024-09-18T11:54:14.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c7fff700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6789/0 conn(0x7fa8c809bcc0 legacy=0x7fa8c811ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 client.?.objecter ms_handle_connect 0x7fa8c809bcc0 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 client.?.objecter ms_handle_connect 0x7fa8c806a830 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8ccee5700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6790/0 conn(0x7fa8c806a830 legacy=0x7fa8c811c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421149152 0 0) 0x7fa8c806efb0 con 0x7fa8c809bcc0 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): my global_id is 94359 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ac004f10 con 0x7fa8c809bcc0 2024-09-18T11:54:14.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 client.?.objecter ms_handle_connect 0x7fa8c8135300 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cd6e6700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.129:6789/0 conn(0x7fa8c8135300 legacy=0x7fa8c8149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120245845 0 0) 0x7fa8c811f310 con 0x7fa8c806a830 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): my global_id is 104234 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c806efb0 con 0x7fa8c806a830 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3777656371 0 0) 0x7fa8ac004f10 con 0x7fa8c809bcc0 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8c811f310 con 0x7fa8c809bcc0 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8b0001c50 con 0x7fa8c809bcc0 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:14.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): dump: 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549099361 0 0) 0x7fa8c8138b40 con 0x7fa8c8135300 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient(hunting): my global_id is 114148 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ac004f10 con 0x7fa8c8135300 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 861175166 0 0) 0x7fa8c811f310 con 0x7fa8c809bcc0 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6790/0 conn(0x7fa8c806a830 legacy=0x7fa8c811c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.129:6789/0 conn(0x7fa8c8135300 legacy=0x7fa8c8149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 monclient: found mon.a 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c811d170 con 0x7fa8c809bcc0 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 5 monclient: authenticate success, global_id 94359 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 10 monclient: _renew_subs 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c811f090 con 0x7fa8c809bcc0 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 10 client.94359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 10 monclient: _renew_subs 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c809b450 con 0x7fa8c809bcc0 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 1 librados: init done 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8b00022b0 con 0x7fa8c809bcc0 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8b0002640 con 0x7fa8c809bcc0 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c6ffd700 4 set_mon_vals no callback set 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 10 monclient: dump: 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa8b00027a0 con 0x7fa8c809bcc0 2024-09-18T11:54:14.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.519+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa8a0006590 con 0x7fa8a0002840 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(380..380 src has 1..380) v4 ==== 5393+0+0 (unknown 4241097072 0 0) 0x7fa8b0043f40 con 0x7fa8c809bcc0 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 client.94359.objecter ms_dispatch 0x7fa8c8000fd0 osd_map(380..380 src has 1..380) v4 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 3 client.94359.objecter handle_osd_map got epochs [380,380] > 0 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 3 client.94359.objecter handle_osd_map decoding full epoch 380 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8cd6e6700 10 monclient: get_auth_request con 0x7fa8ac038420 auth_method 0 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 20 client.94359.objecter dump_active .. 0 homeless 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 client.94359.objecter ms_handle_connect 0x7fa8a0002840 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 client.94359.objecter resend_mon_ops 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8ccee5700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6790/0 conn(0x7fa8a0002840 legacy=0x7fa8a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522765648 0 0) 0x7fa8a0006590 con 0x7fa8a0002840 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 monclient(hunting): my global_id is 104237 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c811f310 con 0x7fa8a0002840 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3458026195 0 0) 0x7fa8c811f310 con 0x7fa8a0002840 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a0006590 con 0x7fa8a0002840 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8b8001a70 con 0x7fa8a0002840 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 monclient: dump: 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.538 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2443884685 0 0) 0x7fa8a0006590 con 0x7fa8a0002840 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8a000a140 con 0x7fa8a0002840 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.523+0000 7fa8c4ff9700 10 client.94359.objecter ms_handle_connect 0x7fa8ac038420 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.527+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fa8b8004830 con 0x7fa8a0002840 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.527+0000 7fa8c4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.527+0000 7fa8c4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.527+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6790/0 conn(0x7fa8a0002840 legacy=0x7fa8a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.527+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6790/0 conn(0x7fa8a0002840 legacy=0x7fa8a0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:14.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.535+0000 7fa8cf149700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:54:14.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.535+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa8a0009670 con 0x7fa8a00090e0 2024-09-18T11:54:14.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8ccee5700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6790/0 conn(0x7fa8a00090e0 legacy=0x7fa8a0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 client.94359.objecter ms_handle_connect 0x7fa8a00090e0 2024-09-18T11:54:14.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 client.94359.objecter resend_mon_ops 2024-09-18T11:54:14.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136147111 0 0) 0x7fa8a0009670 con 0x7fa8a00090e0 2024-09-18T11:54:14.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 monclient(hunting): my global_id is 104240 2024-09-18T11:54:14.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a0006590 con 0x7fa8a00090e0 2024-09-18T11:54:14.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2242301087 0 0) 0x7fa8a0006590 con 0x7fa8a00090e0 2024-09-18T11:54:14.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a0009670 con 0x7fa8a00090e0 2024-09-18T11:54:14.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa8b8004ef0 con 0x7fa8a00090e0 2024-09-18T11:54:14.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:14.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 monclient: dump: 2024-09-18T11:54:14.542 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3237925914 0 0) 0x7fa8a0009670 con 0x7fa8a00090e0 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa8a0009930 con 0x7fa8a00090e0 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 1607007793) 0x7fa8b8005050 con 0x7fa8a00090e0 2024-09-18T11:54:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6790/0 conn(0x7fa8a00090e0 legacy=0x7fa8a0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8c4ff9700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6790/0 conn(0x7fa8a00090e0 legacy=0x7fa8a0008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 46, 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:54:14.544 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.552 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:54:14.553 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.554 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:54:14.555 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:14.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 10 librados: watch_flush enter 2024-09-18T11:54:14.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 10 librados: watch_flush exit 2024-09-18T11:54:14.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 20 client.94359.objecter shutdown clearing up homeless session... 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 10 client.94359.objecter successfully canceled tick 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.129:6812/20806 conn(0x7fa8ac038420 legacy=0x7fa8ac03a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 10 monclient: shutdown 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 >> v1:172.21.15.112:6789/0 conn(0x7fa8c809bcc0 legacy=0x7fa8c811ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cd6e6700 1 -- 172.21.15.112:0/3435450341 reap_dead start 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cd6e6700 1 -- 172.21.15.112:0/3435450341 reap_dead start 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 shutdown_connections 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 >> 172.21.15.112:0/3435450341 conn(0x7fa8c80680d0 msgr2=0x7fa8c809d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 shutdown_connections 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 1 -- 172.21.15.112:0/3435450341 wait complete. 2024-09-18T11:54:14.557 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.539+0000 7fa8cf149700 1 librados: shutdown 2024-09-18T11:54:14.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:54:14.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:54:14.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient: get_monmap_and_config 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient: monmap: 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:14.669486+0000 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:14.669486+0000 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.670 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:14.670 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:14.670 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 1 Processor -- start 2024-09-18T11:54:14.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 1 -- start start 2024-09-18T11:54:14.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient: init 2024-09-18T11:54:14.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:14.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient(hunting): picked mon.noname-a con 0x7fb8e40a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient(hunting): picked mon.noname-c con 0x7fb8e409e370 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient(hunting): picked mon.noname-b con 0x7fb8e4134740 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e406f150 con 0x7fb8e40a2040 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e4096de0 con 0x7fb8e409e370 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e406ee70 con 0x7fb8e4134740 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8ea22f700 10 monclient(hunting): authenticate will time out at 1426.511153s 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8e9a2e700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb8e4134740 0x7fb8e4136b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35606/0 (socket says 172.21.15.112:35606) 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8e922d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb8e40a2040 0x7fb8e412e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54278/0 (socket says 172.21.15.112:54278) 2024-09-18T11:54:14.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.667+0000 7fb8e9a2e700 1 -- 172.21.15.112:0/4161392311 learned_addr learned my addr 172.21.15.112:0/4161392311 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8e922d700 1 -- 172.21.15.112:0/4161392311 >> v1:172.21.15.112:6789/0 conn(0x7fb8e40a2040 legacy=0x7fb8e412e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8e9a2e700 1 -- 172.21.15.112:0/4161392311 >> v1:172.21.15.112:6790/0 conn(0x7fb8e4134740 legacy=0x7fb8e4136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8e8a2c700 1 -- 172.21.15.112:0/4161392311 >> v1:172.21.15.129:6789/0 conn(0x7fb8e409e370 legacy=0x7fb8e409e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909299204 0 0) 0x7fb8e406ee70 con 0x7fb8e4134740 2024-09-18T11:54:14.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): my global_id is 104243 2024-09-18T11:54:14.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d0003570 con 0x7fb8e4134740 2024-09-18T11:54:14.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871056108 0 0) 0x7fb8e406f150 con 0x7fb8e40a2040 2024-09-18T11:54:14.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): my global_id is 94362 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e406ee70 con 0x7fb8e40a2040 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533832621 0 0) 0x7fb8e4096de0 con 0x7fb8e409e370 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient(hunting): my global_id is 114151 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e406f150 con 0x7fb8e409e370 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1380426749 0 0) 0x7fb8d0003570 con 0x7fb8e4134740 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 >> v1:172.21.15.129:6789/0 conn(0x7fb8e409e370 legacy=0x7fb8e409e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 >> v1:172.21.15.112:6789/0 conn(0x7fb8e40a2040 legacy=0x7fb8e412e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 monclient: found mon.noname-b 2024-09-18T11:54:14.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4096b00 con 0x7fb8e4134740 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8ea22f700 5 monclient: authenticate success, global_id 104243 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8ea22f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8dc001d80 con 0x7fb8e4134740 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: dump: 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.677 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8ea22f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8dc001f60 con 0x7fb8e4134740 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8ea22f700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8ea22f700 4 set_mon_vals no callback set 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 1 -- 172.21.15.112:0/4161392311 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8dc004e10 con 0x7fb8e4134740 2024-09-18T11:54:14.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8dbfff700 10 monclient: dump: 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8ea22f700 10 monclient: shutdown 2024-09-18T11:54:14.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8ea22f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.671+0000 7fb8ea22f700 1 -- 172.21.15.112:0/4161392311 >> v1:172.21.15.112:6790/0 conn(0x7fb8e4134740 legacy=0x7fb8e4136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 -- 172.21.15.112:0/4161392311 shutdown_connections 2024-09-18T11:54:14.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 -- 172.21.15.112:0/4161392311 >> 172.21.15.112:0/4161392311 conn(0x7fb8e4067da0 msgr2=0x7fb8e40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 -- 172.21.15.112:0/4161392311 shutdown_connections 2024-09-18T11:54:14.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 -- 172.21.15.112:0/4161392311 wait complete. 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 10 monclient: monmap: 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 librados: starting msgr at 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 librados: starting objecter 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 Processor -- start 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 -- start start 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 librados: setting wanted keys 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 librados: calling monclient init 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 10 monclient: init 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 10 monclient(hunting): picked mon.noname-a con 0x7fb8e409e370 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 10 monclient(hunting): picked mon.noname-c con 0x7fb8e40a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 10 monclient(hunting): picked mon.noname-b con 0x7fb8e4134740 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e4133fa0 con 0x7fb8e409e370 2024-09-18T11:54:14.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8e922d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb8e409e370 0x7fb8e413f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54294/0 (socket says 172.21.15.112:54294) 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e412c920 con 0x7fb8e4134740 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8e922d700 1 -- 172.21.15.112:0/3481995211 learned_addr learned my addr 172.21.15.112:0/3481995211 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.675+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e4134280 con 0x7fb8e40a2040 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 10 monclient(hunting): authenticate will time out at 1426.520027s 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8e922d700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.112:6789/0 conn(0x7fb8e409e370 legacy=0x7fb8e413f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 client.?.objecter ms_handle_connect 0x7fb8e409e370 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8e8a2c700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.112:6790/0 conn(0x7fb8e40a2040 legacy=0x7fb8e409da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 client.?.objecter ms_handle_connect 0x7fb8e40a2040 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003023000 0 0) 0x7fb8e4133fa0 con 0x7fb8e409e370 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): my global_id is 94365 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c8004f40 con 0x7fb8e409e370 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8e9a2e700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6789/0 conn(0x7fb8e4134740 legacy=0x7fb8e4147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 client.?.objecter ms_handle_connect 0x7fb8e4134740 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660543373 0 0) 0x7fb8e4134280 con 0x7fb8e40a2040 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): my global_id is 104246 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e4133fa0 con 0x7fb8e40a2040 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741244665 0 0) 0x7fb8e412c920 con 0x7fb8e4134740 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): my global_id is 114154 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e4134280 con 0x7fb8e4134740 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2856742197 0 0) 0x7fb8c8004f40 con 0x7fb8e409e370 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e412c920 con 0x7fb8e409e370 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8d4001970 con 0x7fb8e409e370 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): dump: 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.689 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1971564596 0 0) 0x7fb8e4133fa0 con 0x7fb8e40a2040 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8c8004f40 con 0x7fb8e40a2040 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8e0001d80 con 0x7fb8e40a2040 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): dump: 2024-09-18T11:54:14.690 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1975137238 0 0) 0x7fb8e4134280 con 0x7fb8e4134740 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e4133fa0 con 0x7fb8e4134740 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8dc004bb0 con 0x7fb8e4134740 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient(hunting): dump: 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1111542837 0 0) 0x7fb8e412c920 con 0x7fb8e409e370 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.112:6790/0 conn(0x7fb8e40a2040 legacy=0x7fb8e409da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6789/0 conn(0x7fb8e4134740 legacy=0x7fb8e4147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 monclient: found mon.a 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e413f770 con 0x7fb8e409e370 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 5 monclient: authenticate success, global_id 94365 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 10 monclient: _renew_subs 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e409e130 con 0x7fb8e409e370 2024-09-18T11:54:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 10 client.94365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 10 monclient: _renew_subs 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e4133040 con 0x7fb8e409e370 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8ea22f700 1 librados: init done 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8d4004ae0 con 0x7fb8e409e370 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8d4004ef0 con 0x7fb8e409e370 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8db7fe700 4 set_mon_vals no callback set 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.679+0000 7fb8d97fa700 10 monclient: dump: 2024-09-18T11:54:14.694 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb8d400f770 con 0x7fb8e409e370 2024-09-18T11:54:14.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(380..380 src has 1..380) v4 ==== 5393+0+0 (unknown 4241097072 0 0) 0x7fb8d4043e90 con 0x7fb8e409e370 2024-09-18T11:54:14.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8d97fa700 10 client.94365.objecter ms_dispatch 0x7fb8e4000fd0 osd_map(380..380 src has 1..380) v4 2024-09-18T11:54:14.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8d97fa700 3 client.94365.objecter handle_osd_map got epochs [380,380] > 0 2024-09-18T11:54:14.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8d97fa700 3 client.94365.objecter handle_osd_map decoding full epoch 380 2024-09-18T11:54:14.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8ea22f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb8e4064d90 con 0x7fb8e402ae80 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8d97fa700 20 client.94365.objecter dump_active .. 0 homeless 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.683+0000 7fb8e8a2c700 10 monclient: get_auth_request con 0x7fb8c8038720 auth_method 0 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 client.94365.objecter ms_handle_connect 0x7fb8e402ae80 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 client.94365.objecter resend_mon_ops 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8e9a2e700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6789/0 conn(0x7fb8e402ae80 legacy=0x7fb8e405e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 client.94365.objecter ms_handle_connect 0x7fb8c8038720 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623856641 0 0) 0x7fb8e4133fa0 con 0x7fb8e402ae80 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 monclient(hunting): my global_id is 114157 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e412c920 con 0x7fb8e402ae80 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3331332754 0 0) 0x7fb8e412c920 con 0x7fb8e402ae80 2024-09-18T11:54:14.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e4133fa0 con 0x7fb8e402ae80 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8dc001900 con 0x7fb8e402ae80 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 monclient: dump: 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.699 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 45391574 0 0) 0x7fb8e4133fa0 con 0x7fb8e402ae80 2024-09-18T11:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8e413ffb0 con 0x7fb8e402ae80 2024-09-18T11:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fb8dc005300 con 0x7fb8e402ae80 2024-09-18T11:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6789/0 conn(0x7fb8e402ae80 legacy=0x7fb8e405e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.687+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6789/0 conn(0x7fb8e402ae80 legacy=0x7fb8e405e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:14.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.699+0000 7fb8ea22f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:54:14.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.699+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb8e406e5d0 con 0x7fb8e4129e70 2024-09-18T11:54:14.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8e9a2e700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6789/0 conn(0x7fb8e4129e70 legacy=0x7fb8e414cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 client.94365.objecter ms_handle_connect 0x7fb8e4129e70 2024-09-18T11:54:14.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 client.94365.objecter resend_mon_ops 2024-09-18T11:54:14.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049585973 0 0) 0x7fb8e4096b00 con 0x7fb8e4129e70 2024-09-18T11:54:14.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 monclient(hunting): my global_id is 114160 2024-09-18T11:54:14.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e4133fa0 con 0x7fb8e4129e70 2024-09-18T11:54:14.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3768781708 0 0) 0x7fb8e4133fa0 con 0x7fb8e4129e70 2024-09-18T11:54:14.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e4096b00 con 0x7fb8e4129e70 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8dc0057e0 con 0x7fb8e4129e70 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 monclient: dump: 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.706 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.707 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.707 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.707 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.707 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1662905716 0 0) 0x7fb8e4096b00 con 0x7fb8e4129e70 2024-09-18T11:54:14.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb8e4064d90 con 0x7fb8e4129e70 2024-09-18T11:54:14.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 4029683487) 0x7fb8dc007dc0 con 0x7fb8e4129e70 2024-09-18T11:54:14.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6789/0 conn(0x7fb8e4129e70 legacy=0x7fb8e414cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8d97fa700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6789/0 conn(0x7fb8e4129e70 legacy=0x7fb8e414cb50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 46, 2024-09-18T11:54:14.708 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.709 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.710 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:54:14.711 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:14.712 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:14.713 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.714 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:14.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.717 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:14.717 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:54:14.717 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:14.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.717 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.717 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:54:14.717 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.718 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:54:14.719 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8ea22f700 10 librados: watch_flush enter 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8ea22f700 10 librados: watch_flush exit 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8ea22f700 20 client.94365.objecter shutdown clearing up homeless session... 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.703+0000 7fb8ea22f700 10 client.94365.objecter successfully canceled tick 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.129:6812/20806 conn(0x7fb8c8038720 legacy=0x7fb8c803abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 10 monclient: shutdown 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 >> v1:172.21.15.112:6789/0 conn(0x7fb8e409e370 legacy=0x7fb8e413f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8e9a2e700 1 -- 172.21.15.112:0/3481995211 reap_dead start 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 shutdown_connections 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 >> 172.21.15.112:0/3481995211 conn(0x7fb8e4067da0 msgr2=0x7fb8e4128290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 shutdown_connections 2024-09-18T11:54:14.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 1 -- 172.21.15.112:0/3481995211 wait complete. 2024-09-18T11:54:14.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.707+0000 7fb8ea22f700 1 librados: shutdown 2024-09-18T11:54:14.725 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:54:14.725 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T11:54:14.725 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-18T11:54:14.725 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T11:54:14.725 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-18T11:54:14.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-18T11:54:14.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient: get_monmap_and_config 2024-09-18T11:54:14.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient: monmap: 2024-09-18T11:54:14.833 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.833 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.833 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:14.832481+0000 2024-09-18T11:54:14.833 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:14.832481+0000 2024-09-18T11:54:14.834 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.834 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.834 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.834 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:14.834 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:14.834 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 1 Processor -- start 2024-09-18T11:54:14.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 1 -- start start 2024-09-18T11:54:14.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient: init 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient(hunting): picked mon.noname-c con 0x7f8c9c067680 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient(hunting): picked mon.noname-b con 0x7f8c9c09db40 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient(hunting): picked mon.noname-a con 0x7f8c9c06f8c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9c099cf0 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9c067a50 con 0x7f8c9c067680 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca1cf6700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8c9c09db40 0x7f8c9c09d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35632/0 (socket says 172.21.15.112:35632) 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca1cf6700 1 -- 172.21.15.112:0/1776912180 learned_addr learned my addr 172.21.15.112:0/1776912180 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 1 -- 172.21.15.112:0/1776912180 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9c09d8a0 con 0x7f8c9c09db40 2024-09-18T11:54:14.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.831+0000 7f8ca475b700 10 monclient(hunting): authenticate will time out at 1426.674699s 2024-09-18T11:54:14.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca1cf6700 1 -- 172.21.15.112:0/1776912180 >> v1:172.21.15.112:6790/0 conn(0x7f8c9c09db40 legacy=0x7f8c9c09d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca2cf8700 1 -- 172.21.15.112:0/1776912180 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c06f8c0 legacy=0x7f8c9c1388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca24f7700 1 -- 172.21.15.112:0/1776912180 >> v1:172.21.15.129:6789/0 conn(0x7f8c9c067680 legacy=0x7f8c9c09ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582711233 0 0) 0x7f8c9c09d8a0 con 0x7f8c9c09db40 2024-09-18T11:54:14.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient(hunting): my global_id is 104249 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c88003570 con 0x7f8c9c09db40 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327898400 0 0) 0x7f8c9c099cf0 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient(hunting): my global_id is 94368 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c09d8a0 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725613548 0 0) 0x7f8c9c09d8a0 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 >> v1:172.21.15.112:6790/0 conn(0x7f8c9c09db40 legacy=0x7f8c9c09d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 >> v1:172.21.15.129:6789/0 conn(0x7f8c9c067680 legacy=0x7f8c9c09ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 monclient: found mon.noname-a 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c099980 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8c94001cf0 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: dump: 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.843 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 5 monclient: authenticate success, global_id 94368 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c94001f80 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 1 -- 172.21.15.112:0/1776912180 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8c94002420 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 4 set_mon_vals no callback set 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca14f5700 10 monclient: dump: 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.845 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 10 monclient: shutdown 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 -- 172.21.15.112:0/1776912180 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c06f8c0 legacy=0x7f8c9c1388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 -- 172.21.15.112:0/1776912180 shutdown_connections 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 -- 172.21.15.112:0/1776912180 >> 172.21.15.112:0/1776912180 conn(0x7f8c9c068b30 msgr2=0x7f8c9c0a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 -- 172.21.15.112:0/1776912180 shutdown_connections 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 -- 172.21.15.112:0/1776912180 wait complete. 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 10 monclient: build_initial_monmap 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 10 monclient: monmap: 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 librados: starting msgr at 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 librados: starting objecter 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 Processor -- start 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 -- start start 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 librados: setting wanted keys 2024-09-18T11:54:14.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 1 librados: calling monclient init 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.835+0000 7f8ca475b700 10 monclient: init 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient(hunting): picked mon.noname-a con 0x7f8c9c067680 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient(hunting): picked mon.noname-c con 0x7f8c9c06f8c0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient(hunting): picked mon.noname-b con 0x7f8c9c09db40 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9c132b50 con 0x7f8c9c067680 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9c139830 con 0x7f8c9c09db40 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9c140020 con 0x7f8c9c06f8c0 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient(hunting): authenticate will time out at 1426.680322s 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca1cf6700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8c9c06f8c0 0x7f8c9c144510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35636/0 (socket says 172.21.15.112:35636) 2024-09-18T11:54:14.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca24f7700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8c9c067680 0x7f8c9c130ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:54310/0 (socket says 172.21.15.112:54310) 2024-09-18T11:54:14.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca1cf6700 1 -- 172.21.15.112:0/1509046137 learned_addr learned my addr 172.21.15.112:0/1509046137 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca1cf6700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6790/0 conn(0x7f8c9c06f8c0 legacy=0x7f8c9c144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 client.?.objecter ms_handle_connect 0x7f8c9c06f8c0 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca24f7700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c067680 legacy=0x7f8c9c130ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 client.?.objecter ms_handle_connect 0x7f8c9c067680 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 client.?.objecter ms_handle_connect 0x7f8c9c09db40 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca2cf8700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.129:6789/0 conn(0x7f8c9c09db40 legacy=0x7f8c9c147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288492182 0 0) 0x7f8c9c132b50 con 0x7f8c9c067680 2024-09-18T11:54:14.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): my global_id is 94371 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c80004f10 con 0x7f8c9c067680 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929954722 0 0) 0x7f8c9c139830 con 0x7f8c9c09db40 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): my global_id is 114166 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c132b50 con 0x7f8c9c09db40 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1149438909 0 0) 0x7f8c80004f10 con 0x7f8c9c067680 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c9c139830 con 0x7f8c9c067680 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8c98001bc0 con 0x7f8c9c067680 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:54:14.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): dump: 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3106367329 0 0) 0x7f8c9c139830 con 0x7f8c9c067680 2024-09-18T11:54:14.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6790/0 conn(0x7f8c9c06f8c0 legacy=0x7f8c9c144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.129:6789/0 conn(0x7f8c9c09db40 legacy=0x7f8c9c147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient: _finish_hunting 0 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 monclient: found mon.a 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c137a20 con 0x7f8c9c067680 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 10 monclient: _finish_auth 0 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8c927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 5 monclient: authenticate success, global_id 94371 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient: _renew_subs 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c9c139550 con 0x7f8c9c067680 2024-09-18T11:54:14.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 client.94371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient: _renew_subs 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c9c1385f0 con 0x7f8c9c067680 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.839+0000 7f8ca475b700 1 librados: init done 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8ca475b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8c9c1399f0 con 0x7f8c9c136f90 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c980022b0 con 0x7f8c9c067680 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8c98002600 con 0x7f8c9c067680 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient: dump: 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.861 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8c980027e0 con 0x7f8c9c067680 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8ca0cf4700 4 set_mon_vals no callback set 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(381..381 src has 1..381) v4 ==== 5409+0+0 (unknown 1825515518 0 0) 0x7f8c98043eb0 con 0x7f8c9c067680 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 client.94371.objecter ms_dispatch 0x7f8c9c000fd0 osd_map(381..381 src has 1..381) v4 2024-09-18T11:54:14.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 3 client.94371.objecter handle_osd_map got epochs [381,381] > 0 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 3 client.94371.objecter handle_osd_map decoding full epoch 381 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 20 client.94371.objecter dump_active .. 0 homeless 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8ca2cf8700 10 monclient: get_auth_request con 0x7f8c80038360 auth_method 0 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8ca1cf6700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c136f90 legacy=0x7f8c9c06cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 client.94371.objecter ms_handle_connect 0x7f8c9c136f90 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 client.94371.objecter resend_mon_ops 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700455473 0 0) 0x7f8c9c1399f0 con 0x7f8c9c136f90 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient(hunting): my global_id is 94374 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c139830 con 0x7f8c9c136f90 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3886536932 0 0) 0x7f8c9c139830 con 0x7f8c9c136f90 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c9c1399f0 con 0x7f8c9c136f90 2024-09-18T11:54:14.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8c8c001a90 con 0x7f8c9c136f90 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 10 monclient: dump: 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.864 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1841209972 0 0) 0x7f8c9c1399f0 con 0x7f8c9c136f90 2024-09-18T11:54:14.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.843+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c9c137360 con 0x7f8c9c136f90 2024-09-18T11:54:14.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.847+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f8c8c005920 con 0x7f8c9c136f90 2024-09-18T11:54:14.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.847+0000 7f8c927fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:14.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.847+0000 7f8c927fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:14.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.847+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c136f90 legacy=0x7f8c9c06cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.847+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c136f90 legacy=0x7f8c9c06cf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:14.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.847+0000 7f8c927fc700 10 client.94371.objecter ms_handle_connect 0x7f8c80038360 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.855+0000 7f8ca475b700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.855+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8c9c05f2e0 con 0x7f8c9c02ad10 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8ca1cf6700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c02ad10 legacy=0x7f8c9c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 client.94371.objecter ms_handle_connect 0x7f8c9c02ad10 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 client.94371.objecter resend_mon_ops 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554376719 0 0) 0x7f8c9c05f2e0 con 0x7f8c9c02ad10 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:14.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 monclient(hunting): my global_id is 94377 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c1399f0 con 0x7f8c9c02ad10 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 180557052 0 0) 0x7f8c9c1399f0 con 0x7f8c9c02ad10 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c9c05f2e0 con 0x7f8c9c02ad10 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8c8c004cc0 con 0x7f8c9c02ad10 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 monclient: dump: 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:14.868 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4182741270 0 0) 0x7f8c9c05f2e0 con 0x7f8c9c02ad10 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7f8c9c064f00 con 0x7f8c9c02ad10 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7f8c8c005a90 con 0x7f8c9c02ad10 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c02ad10 legacy=0x7f8c9c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8c927fc700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c02ad10 legacy=0x7f8c9c06d690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:14.870 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:14.870 INFO:teuthology.orchestra.run.smithi112.stdout: "ret": 0, 2024-09-18T11:54:14.870 INFO:teuthology.orchestra.run.smithi112.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-18T11:54:14.870 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8ca475b700 10 librados: watch_flush enter 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8ca475b700 10 librados: watch_flush exit 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8ca475b700 20 client.94371.objecter shutdown clearing up homeless session... 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8ca475b700 10 client.94371.objecter successfully canceled tick 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.859+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.129:6812/20806 conn(0x7f8c80038360 legacy=0x7f8c8003a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca475b700 10 monclient: shutdown 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca475b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 >> v1:172.21.15.112:6789/0 conn(0x7f8c9c067680 legacy=0x7f8c9c130ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:14.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca2cf8700 1 -- 172.21.15.112:0/1509046137 reap_dead start 2024-09-18T11:54:14.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca2cf8700 1 -- 172.21.15.112:0/1509046137 reap_dead start 2024-09-18T11:54:14.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 shutdown_connections 2024-09-18T11:54:14.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 >> 172.21.15.112:0/1509046137 conn(0x7f8c9c068b30 msgr2=0x7f8c9c0a1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:14.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 shutdown_connections 2024-09-18T11:54:14.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca475b700 1 -- 172.21.15.112:0/1509046137 wait complete. 2024-09-18T11:54:14.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:14.863+0000 7f8ca475b700 1 librados: shutdown 2024-09-18T11:54:14.880 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T11:54:14.880 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T11:54:14.880 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:54:14.893 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T11:54:14.893 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T11:54:14.893 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T11:54:14.893 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T11:54:29.894 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T11:54:29.895 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T11:54:29.895 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T11:54:29.895 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:54:30.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient: get_monmap_and_config 2024-09-18T11:54:30.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient: build_initial_monmap 2024-09-18T11:54:30.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient: monmap: 2024-09-18T11:54:30.005 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:30.005 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:30.005 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:30.004812+0000 2024-09-18T11:54:30.005 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:30.004812+0000 2024-09-18T11:54:30.005 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:30.005 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:30.005 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:30.005 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:30.008 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:30.009 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:30.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 1 Processor -- start 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 1 -- start start 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient: init 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient(hunting): picked mon.noname-c con 0x7fb21806aa40 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient(hunting): picked mon.noname-a con 0x7fb2180a1e80 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient(hunting): picked mon.noname-b con 0x7fb2181386e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218097270 con 0x7fb2180a1e80 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb21db44700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb2180a1160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21806ae10 con 0x7fb21806aa40 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218097550 con 0x7fb2181386e0 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:30.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb2205a9700 10 monclient(hunting): authenticate will time out at 1441.846422s 2024-09-18T11:54:30.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb21eb46700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb2181386e0 0x7fb21813aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50676/0 (socket says 172.21.15.112:50676) 2024-09-18T11:54:30.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.003+0000 7fb21eb46700 1 -- 172.21.15.112:0/3457620 learned_addr learned my addr 172.21.15.112:0/3457620 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:30.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.007+0000 7fb21eb46700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6790/0 conn(0x7fb2181386e0 legacy=0x7fb21813aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:30.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.007+0000 7fb21e345700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.129:6789/0 conn(0x7fb21806aa40 legacy=0x7fb2180a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:30.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.207+0000 7fb21db44700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb2180a1160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:30.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:30.611+0000 7fb21db44700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb2180a1160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:31.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:31.415+0000 7fb21db44700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb2180a1160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:33.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 10 monclient(hunting): tick 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 1 monclient(hunting): continuing hunt 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6790/0 conn(0x7fb2181386e0 legacy=0x7fb21813aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.129:6789/0 conn(0x7fb21806aa40 legacy=0x7fb2180a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb2180a1160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fb2180a1e80 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fb21806aa40 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fb2181386e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 1 -- 172.21.15.112:0/3457620 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218096a60 con 0x7fb2180a1e80 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb21e345700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb204005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 1 -- 172.21.15.112:0/3457620 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218097270 con 0x7fb21806aa40 2024-09-18T11:54:33.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 1 -- 172.21.15.112:0/3457620 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2040100d0 con 0x7fb2181386e0 2024-09-18T11:54:33.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.003+0000 7fb20ffff700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:33.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.007+0000 7fb21eb46700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6790/0 conn(0x7fb2181386e0 legacy=0x7fb20400c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:33.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.007+0000 7fb21db44700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.129:6789/0 conn(0x7fb21806aa40 legacy=0x7fb204009020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:33.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.207+0000 7fb21e345700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb204005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:33.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:33.611+0000 7fb21e345700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb204005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:34.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.415+0000 7fb21e345700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb204005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:34.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.931+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642447844 0 0) 0x7fb218097270 con 0x7fb21806aa40 2024-09-18T11:54:34.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.931+0000 7fb21d343700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:34.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.931+0000 7fb21d343700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:34.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.931+0000 7fb21d343700 10 monclient(hunting): my global_id is 114169 2024-09-18T11:54:34.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.931+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb200003570 con 0x7fb21806aa40 2024-09-18T11:54:34.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 664784204 0 0) 0x7fb200003570 con 0x7fb21806aa40 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6790/0 conn(0x7fb2181386e0 legacy=0x7fb20400c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb204005860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 10 monclient: _finish_hunting 0 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 1 monclient: found mon.noname-c 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204011330 con 0x7fb21806aa40 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 10 monclient: _finish_auth 0 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb210004830 con 0x7fb21806aa40 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb2205a9700 5 monclient: authenticate success, global_id 114169 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb2205a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb21d343700 10 monclient: dump: 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:34.939 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:34.940 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.935+0000 7fb2205a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:34.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.939+0000 7fb21db44700 1 --1- 172.21.15.112:0/3457620 >> v1:172.21.15.129:6789/0 conn(0x7fb21806aa40 0x7fb204009020 :-1 s=OPENED pgs=9053 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7fb210004b00 config(0 keys) v1 2024-09-18T11:54:34.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.939+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb210004b00 con 0x7fb21806aa40 2024-09-18T11:54:34.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.939+0000 7fb21d343700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:34.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.939+0000 7fb2205a9700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:34.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.939+0000 7fb2205a9700 4 set_mon_vals no callback set 2024-09-18T11:54:34.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 10 monclient: shutdown 2024-09-18T11:54:34.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:34.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 -- 172.21.15.112:0/3457620 >> v1:172.21.15.129:6789/0 conn(0x7fb21806aa40 legacy=0x7fb204009020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:34.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb21d343700 1 -- 172.21.15.112:0/3457620 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb210004e10 con 0x7fb21806aa40 2024-09-18T11:54:34.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb21d343700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T11:54:34.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 -- 172.21.15.112:0/3457620 shutdown_connections 2024-09-18T11:54:34.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 -- 172.21.15.112:0/3457620 >> 172.21.15.112:0/3457620 conn(0x7fb2180a5d10 msgr2=0x7fb2180a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:34.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 -- 172.21.15.112:0/3457620 shutdown_connections 2024-09-18T11:54:34.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 -- 172.21.15.112:0/3457620 wait complete. 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 10 monclient: build_initial_monmap 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 10 monclient: monmap: 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:34.948 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:34.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 librados: starting msgr at 2024-09-18T11:54:34.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 librados: starting objecter 2024-09-18T11:54:34.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 Processor -- start 2024-09-18T11:54:34.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 -- start start 2024-09-18T11:54:34.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 librados: setting wanted keys 2024-09-18T11:54:34.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 1 librados: calling monclient init 2024-09-18T11:54:34.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.943+0000 7fb2205a9700 10 monclient: init 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 10 monclient(hunting): picked mon.noname-b con 0x7fb21806aa40 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 10 monclient(hunting): picked mon.noname-a con 0x7fb2180a1e80 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 10 monclient(hunting): picked mon.noname-c con 0x7fb2181386e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2180a22f0 con 0x7fb2180a1e80 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2180a5380 con 0x7fb21806aa40 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218121330 con 0x7fb2181386e0 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb2205a9700 10 monclient(hunting): authenticate will time out at 1446.789089s 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb21db44700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb21811f3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb21eb46700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb2181386e0 0x7fb21811fad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50724/0 (socket says 172.21.15.112:50724) 2024-09-18T11:54:34.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb21eb46700 1 -- 172.21.15.112:0/2687057823 learned_addr learned my addr 172.21.15.112:0/2687057823 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:34.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 10 client.?.objecter ms_handle_connect 0x7fb2181386e0 2024-09-18T11:54:34.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb21eb46700 1 -- 172.21.15.112:0/2687057823 >> v1:172.21.15.112:6790/0 conn(0x7fb2181386e0 legacy=0x7fb21811fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:34.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:34.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 10 client.?.objecter ms_handle_connect 0x7fb21806aa40 2024-09-18T11:54:34.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:34.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb21e345700 1 -- 172.21.15.112:0/2687057823 >> v1:172.21.15.129:6789/0 conn(0x7fb21806aa40 legacy=0x7fb21811ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:34.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549165496 0 0) 0x7fb2180a5380 con 0x7fb21806aa40 2024-09-18T11:54:34.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:34.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:34.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 10 monclient(hunting): my global_id is 114172 2024-09-18T11:54:34.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.947+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb204009f20 con 0x7fb21806aa40 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1032728510 0 0) 0x7fb204009f20 con 0x7fb21806aa40 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2180a5380 con 0x7fb21806aa40 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb214001d20 con 0x7fb21806aa40 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient(hunting): dump: 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:34.954 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4057694206 0 0) 0x7fb2180a5380 con 0x7fb21806aa40 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 >> v1:172.21.15.112:6790/0 conn(0x7fb2181386e0 legacy=0x7fb21811fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 >> v1:172.21.15.112:6789/0 conn(0x7fb2180a1e80 legacy=0x7fb21811f3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 monclient: found mon.b 2024-09-18T11:54:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218122290 con 0x7fb21806aa40 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 5 monclient: authenticate success, global_id 114172 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 10 monclient: _renew_subs 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb218120760 con 0x7fb21806aa40 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 10 client.114172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 10 monclient: _renew_subs 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:54:34.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21814aba0 con 0x7fb21806aa40 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 1 librados: init done 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb214002330 con 0x7fb21806aa40 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb2140026c0 con 0x7fb21806aa40 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 monclient: dump: 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:34.958 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb21cb42700 4 set_mon_vals no callback set 2024-09-18T11:54:34.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:34.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:34.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:54:34.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f80050b0 con 0x7fb21806aa40 2024-09-18T11:54:34.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb214002a10 con 0x7fb21806aa40 2024-09-18T11:54:34.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(381..381 src has 1..381) v4 ==== 5409+0+0 (unknown 1825515518 0 0) 0x7fb214044090 con 0x7fb21806aa40 2024-09-18T11:54:34.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 10 client.114172.objecter ms_dispatch 0x7fb218000fd0 osd_map(381..381 src has 1..381) v4 2024-09-18T11:54:34.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 3 client.114172.objecter handle_osd_map got epochs [381,381] > 0 2024-09-18T11:54:34.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 3 client.114172.objecter handle_osd_map decoding full epoch 381 2024-09-18T11:54:34.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb20e7fc700 20 client.114172.objecter dump_active .. 0 homeless 2024-09-18T11:54:34.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.951+0000 7fb21db44700 10 monclient: get_auth_request con 0x7fb204038590 auth_method 0 2024-09-18T11:54:34.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.955+0000 7fb20e7fc700 10 client.114172.objecter ms_handle_connect 0x7fb204038590 2024-09-18T11:54:34.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.959+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fb214014c50 con 0x7fb21806aa40 2024-09-18T11:54:34.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.959+0000 7fb20e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:34.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:34.959+0000 7fb20e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:35.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.179+0000 7fb2205a9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:54:35.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.179+0000 7fb2205a9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:54:35.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.179+0000 7fb2205a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:54:35.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.179+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb1f80058e0 con 0x7fb21806aa40 2024-09-18T11:54:35.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.179+0000 7fb20e7fc700 1 -- 172.21.15.112:0/2687057823 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1716287142) 0x7fb214014650 con 0x7fb21806aa40 2024-09-18T11:54:35.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.179+0000 7fb20e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:54:35.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.179+0000 7fb20e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:35.183 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":48,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:54:35.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 10 librados: watch_flush enter 2024-09-18T11:54:35.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 10 librados: watch_flush exit 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 20 client.114172.objecter shutdown clearing up homeless session... 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 10 client.114172.objecter successfully canceled tick 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 >> v1:172.21.15.129:6812/20806 conn(0x7fb204038590 legacy=0x7fb20403aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 10 monclient: shutdown 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 >> v1:172.21.15.129:6789/0 conn(0x7fb21806aa40 legacy=0x7fb21811ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 shutdown_connections 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 >> 172.21.15.112:0/2687057823 conn(0x7fb2180a5d10 msgr2=0x7fb21809ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 shutdown_connections 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 1 -- 172.21.15.112:0/2687057823 wait complete. 2024-09-18T11:54:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.183+0000 7fb2205a9700 1 librados: shutdown 2024-09-18T11:54:35.196 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T11:54:35.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T11:54:35.197 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient: get_monmap_and_config 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient: build_initial_monmap 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient: monmap: 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:35.301381+0000 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:35.301381+0000 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:35.301 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:35.302 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 1 Processor -- start 2024-09-18T11:54:35.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 1 -- start start 2024-09-18T11:54:35.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient: init 2024-09-18T11:54:35.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:35.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:35.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient(hunting): picked mon.noname-a con 0x7f1b8c0a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient(hunting): picked mon.noname-c con 0x7f1b8c0a0750 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient(hunting): picked mon.noname-b con 0x7f1b8c1385b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c097270 con 0x7f1b8c0a0380 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0a3170 con 0x7f1b8c0a0750 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c096cb0 con 0x7f1b8c1385b0 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b90c4d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f1b8c0a0380 legacy=0x7f1b8c097970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b91c4f700 10 monclient(hunting): authenticate will time out at 1447.143028s 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b9144e700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1b8c1385b0 0x7f1b8c13a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50730/0 (socket says 172.21.15.112:50730) 2024-09-18T11:54:35.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.299+0000 7f1b9144e700 1 -- 172.21.15.112:0/3345037228 learned_addr learned my addr 172.21.15.112:0/3345037228 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:35.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b9144e700 1 -- 172.21.15.112:0/3345037228 >> v1:172.21.15.112:6790/0 conn(0x7f1b8c1385b0 legacy=0x7f1b8c13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588438940 0 0) 0x7f1b8c096cb0 con 0x7f1b8c1385b0 2024-09-18T11:54:35.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient(hunting): my global_id is 104252 2024-09-18T11:54:35.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b78003570 con 0x7f1b8c1385b0 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8bfff700 1 -- 172.21.15.112:0/3345037228 >> v1:172.21.15.129:6789/0 conn(0x7f1b8c0a0750 legacy=0x7f1b8c0a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570238243 0 0) 0x7f1b78003570 con 0x7f1b8c1385b0 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 >> v1:172.21.15.129:6789/0 conn(0x7f1b8c0a0750 legacy=0x7f1b8c0a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 >> v1:172.21.15.112:6789/0 conn(0x7f1b8c0a0380 legacy=0x7f1b8c097970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 monclient: found mon.noname-b 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c0a1420 con 0x7f1b8c1385b0 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: _finish_auth 0 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:35.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 5 monclient: authenticate success, global_id 104252 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1b84001cc0 con 0x7f1b8c1385b0 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: dump: 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.309 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b84001f90 con 0x7f1b8c1385b0 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 1 -- 172.21.15.112:0/3345037228 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1b84004e10 con 0x7f1b8c1385b0 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 4 set_mon_vals no callback set 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b8b7fe700 10 monclient: dump: 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.310 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 10 monclient: shutdown 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 1 -- 172.21.15.112:0/3345037228 >> v1:172.21.15.112:6790/0 conn(0x7f1b8c1385b0 legacy=0x7f1b8c13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 1 -- 172.21.15.112:0/3345037228 shutdown_connections 2024-09-18T11:54:35.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 1 -- 172.21.15.112:0/3345037228 >> 172.21.15.112:0/3345037228 conn(0x7f1b8c0a5990 msgr2=0x7f1b8c0a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:35.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 1 -- 172.21.15.112:0/3345037228 shutdown_connections 2024-09-18T11:54:35.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.303+0000 7f1b91c4f700 1 -- 172.21.15.112:0/3345037228 wait complete. 2024-09-18T11:54:35.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient: build_initial_monmap 2024-09-18T11:54:35.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient: monmap: 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 librados: starting msgr at 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 librados: starting objecter 2024-09-18T11:54:35.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 Processor -- start 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 -- start start 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 librados: setting wanted keys 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 librados: calling monclient init 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient: init 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient(hunting): picked mon.noname-a con 0x7f1b8c0a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient(hunting): picked mon.noname-c con 0x7f1b8c0a0750 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient(hunting): picked mon.noname-b con 0x7f1b8c1385b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c09eeb0 con 0x7f1b8c0a0380 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c1370e0 con 0x7f1b8c1385b0 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c097f50 con 0x7f1b8c0a0750 2024-09-18T11:54:35.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b91c4f700 10 monclient(hunting): authenticate will time out at 1447.150092s 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b90c4d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f1b8c0a0380 legacy=0x7f1b8c141630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b8bfff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1b8c0a0750 0x7f1b8c09dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50746/0 (socket says 172.21.15.112:50746) 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.307+0000 7f1b8bfff700 1 -- 172.21.15.112:0/2374391666 learned_addr learned my addr 172.21.15.112:0/2374391666 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 client.?.objecter ms_handle_connect 0x7f1b8c1385b0 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b9144e700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.129:6789/0 conn(0x7f1b8c1385b0 legacy=0x7f1b8c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 client.?.objecter ms_handle_connect 0x7f1b8c0a0750 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b8bfff700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6790/0 conn(0x7f1b8c0a0750 legacy=0x7f1b8c09dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690191219 0 0) 0x7f1b8c097f50 con 0x7f1b8c0a0750 2024-09-18T11:54:35.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): my global_id is 104255 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c004f10 con 0x7f1b8c0a0750 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945857055 0 0) 0x7f1b8c1370e0 con 0x7f1b8c1385b0 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): my global_id is 114178 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b8c097f50 con 0x7f1b8c1385b0 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 410214612 0 0) 0x7f1b6c004f10 con 0x7f1b8c0a0750 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b8c1370e0 con 0x7f1b8c0a0750 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1b7c001ca0 con 0x7f1b8c0a0750 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:54:35.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): dump: 2024-09-18T11:54:35.322 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.322 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.322 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.322 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.322 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.325 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 169407540 0 0) 0x7f1b8c1370e0 con 0x7f1b8c0a0750 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.129:6789/0 conn(0x7f1b8c1385b0 legacy=0x7f1b8c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6789/0 conn(0x7f1b8c0a0380 legacy=0x7f1b8c141630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 monclient: found mon.c 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c098230 con 0x7f1b8c0a0750 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient: _finish_auth 0 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 5 monclient: authenticate success, global_id 104255 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 10 monclient: _renew_subs 2024-09-18T11:54:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c141d40 con 0x7f1b8c0a0750 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 10 client.104255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 10 monclient: _renew_subs 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c09e510 con 0x7f1b8c0a0750 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 1 librados: init done 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b7c0022b0 con 0x7f1b8c0a0750 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b8affd700 4 set_mon_vals no callback set 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1b7c0026c0 con 0x7f1b8c0a0750 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 monclient: dump: 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1b70006590 con 0x7f1b70002840 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f1b7c0028a0 con 0x7f1b8c0a0750 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(381..381 src has 1..381) v4 ==== 5409+0+0 (unknown 1825515518 0 0) 0x7f1b7c043fd0 con 0x7f1b8c0a0750 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 10 client.104255.objecter ms_dispatch 0x7f1b8c000fd0 osd_map(381..381 src has 1..381) v4 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 3 client.104255.objecter handle_osd_map got epochs [381,381] > 0 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 3 client.104255.objecter handle_osd_map decoding full epoch 381 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.311+0000 7f1b88ff9700 20 client.104255.objecter dump_active .. 0 homeless 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b9144e700 10 monclient: get_auth_request con 0x7f1b6c038070 auth_method 0 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 client.104255.objecter ms_handle_connect 0x7f1b70002840 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 client.104255.objecter resend_mon_ops 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b90c4d700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6790/0 conn(0x7f1b70002840 legacy=0x7f1b70004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647320483 0 0) 0x7f1b70006590 con 0x7f1b70002840 2024-09-18T11:54:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 monclient(hunting): my global_id is 104258 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b8c1370e0 con 0x7f1b70002840 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3580065433 0 0) 0x7f1b8c1370e0 con 0x7f1b70002840 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b70006590 con 0x7f1b70002840 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1b740043b0 con 0x7f1b70002840 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 monclient: dump: 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.331 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 823935661 0 0) 0x7f1b70006590 con 0x7f1b70002840 2024-09-18T11:54:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b7000a140 con 0x7f1b70002840 2024-09-18T11:54:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 client.104255.objecter ms_handle_connect 0x7f1b6c038070 2024-09-18T11:54:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f1b74006690 con 0x7f1b70002840 2024-09-18T11:54:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6790/0 conn(0x7f1b70002840 legacy=0x7f1b70004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.315+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6790/0 conn(0x7f1b70002840 legacy=0x7f1b70004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.323+0000 7f1b91c4f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.323+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1b70009670 con 0x7f1b700090e0 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b90c4d700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6790/0 conn(0x7f1b700090e0 legacy=0x7f1b70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 client.104255.objecter ms_handle_connect 0x7f1b700090e0 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 client.104255.objecter resend_mon_ops 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982413288 0 0) 0x7f1b70009670 con 0x7f1b700090e0 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 monclient(hunting): my global_id is 104261 2024-09-18T11:54:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b70006590 con 0x7f1b700090e0 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4249477695 0 0) 0x7f1b70006590 con 0x7f1b700090e0 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b70009670 con 0x7f1b700090e0 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1b74004b90 con 0x7f1b700090e0 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 monclient: dump: 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3384549667 0 0) 0x7f1b70009670 con 0x7f1b700090e0 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1b70009930 con 0x7f1b700090e0 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3990 (unknown 0 0 278276153) 0x7f1b74006920 con 0x7f1b700090e0 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6790/0 conn(0x7f1b700090e0 legacy=0x7f1b70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b88ff9700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6790/0 conn(0x7f1b700090e0 legacy=0x7f1b70008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:35.336 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 48, 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:54:35.337 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:35.338 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:54:35.339 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:54:35.340 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.341 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.342 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:54:35.343 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:35.344 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:35.345 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.346 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.347 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.348 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:54:35.348 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:35.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b91c4f700 10 librados: watch_flush enter 2024-09-18T11:54:35.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b91c4f700 10 librados: watch_flush exit 2024-09-18T11:54:35.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b91c4f700 20 client.104255.objecter shutdown clearing up homeless session... 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b91c4f700 10 client.104255.objecter successfully canceled tick 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.327+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.129:6812/20806 conn(0x7f1b6c038070 legacy=0x7f1b6c03a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b91c4f700 10 monclient: shutdown 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b91c4f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 >> v1:172.21.15.112:6790/0 conn(0x7f1b8c0a0750 legacy=0x7f1b8c09dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b9144e700 1 -- 172.21.15.112:0/2374391666 reap_dead start 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b9144e700 1 -- 172.21.15.112:0/2374391666 reap_dead start 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 shutdown_connections 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 >> 172.21.15.112:0/2374391666 conn(0x7f1b8c0a5990 msgr2=0x7f1b8c138980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 shutdown_connections 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b91c4f700 1 -- 172.21.15.112:0/2374391666 wait complete. 2024-09-18T11:54:35.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.331+0000 7f1b91c4f700 1 librados: shutdown 2024-09-18T11:54:35.352 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:54:35.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T11:54:35.352 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:54:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.455+0000 7fe6cfb37700 10 monclient: get_monmap_and_config 2024-09-18T11:54:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.455+0000 7fe6cfb37700 10 monclient: build_initial_monmap 2024-09-18T11:54:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.455+0000 7fe6cfb37700 10 monclient: monmap: 2024-09-18T11:54:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:54:35.459061+0000 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:54:35.459061+0000 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:54:35.459 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.455+0000 7fe6cfb37700 1 Processor -- start 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.455+0000 7fe6cfb37700 1 -- start start 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.455+0000 7fe6cfb37700 10 monclient: init 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 10 monclient(hunting): picked mon.noname-b con 0x7fe6c80a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 10 monclient(hunting): picked mon.noname-a con 0x7fe6c806e9d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 10 monclient(hunting): picked mon.noname-c con 0x7fe6c8135300 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c80a1600 con 0x7fe6c806e9d0 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c806eda0 con 0x7fe6c8135300 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c806b5a0 con 0x7fe6c80a1230 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 10 monclient(hunting): authenticate will time out at 1447.300541s 2024-09-18T11:54:35.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cd0d2700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe6c806e9d0 legacy=0x7fe6c806ae90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:35.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cd8d3700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe6c80a1230 0x7fe6c806a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50780/0 (socket says 172.21.15.112:50780) 2024-09-18T11:54:35.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cd8d3700 1 -- 172.21.15.112:0/1579717622 learned_addr learned my addr 172.21.15.112:0/1579717622 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:35.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cd8d3700 1 -- 172.21.15.112:0/1579717622 >> v1:172.21.15.112:6790/0 conn(0x7fe6c80a1230 legacy=0x7fe6c806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602529761 0 0) 0x7fe6c806b5a0 con 0x7fe6c80a1230 2024-09-18T11:54:35.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient(hunting): my global_id is 104264 2024-09-18T11:54:35.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b4003570 con 0x7fe6c80a1230 2024-09-18T11:54:35.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3246117472 0 0) 0x7fe6b4003570 con 0x7fe6c80a1230 2024-09-18T11:54:35.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 >> v1:172.21.15.129:6789/0 conn(0x7fe6c8135300 legacy=0x7fe6c81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T11:54:35.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 >> v1:172.21.15.112:6789/0 conn(0x7fe6c806e9d0 legacy=0x7fe6c806ae90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:54:35.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient: _finish_hunting 0 2024-09-18T11:54:35.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 1 monclient: found mon.noname-b 2024-09-18T11:54:35.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:35.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:54:35.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c809f9e0 con 0x7fe6c80a1230 2024-09-18T11:54:35.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient: _finish_auth 0 2024-09-18T11:54:35.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 5 monclient: authenticate success, global_id 104264 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6bc001940 con 0x7fe6c80a1230 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cc8d1700 10 monclient: dump: 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.467 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.459+0000 7fe6cfb37700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6bc001c10 con 0x7fe6c80a1230 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cc8d1700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient: get_monmap_and_config success 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 4 set_mon_vals no callback set 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cc8d1700 1 -- 172.21.15.112:0/1579717622 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6bc004a90 con 0x7fe6c80a1230 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cc8d1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cc8d1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cc8d1700 10 monclient: dump: 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.468 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.469 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.469 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.469 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.469 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.469 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient: shutdown 2024-09-18T11:54:35.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:35.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- 172.21.15.112:0/1579717622 >> v1:172.21.15.112:6790/0 conn(0x7fe6c80a1230 legacy=0x7fe6c806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- 172.21.15.112:0/1579717622 shutdown_connections 2024-09-18T11:54:35.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- 172.21.15.112:0/1579717622 >> 172.21.15.112:0/1579717622 conn(0x7fe6c80680d0 msgr2=0x7fe6c80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- 172.21.15.112:0/1579717622 shutdown_connections 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- 172.21.15.112:0/1579717622 wait complete. 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient: build_initial_monmap 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient: monmap: 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.476 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 librados: starting msgr at 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 librados: starting objecter 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 Processor -- start 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- start start 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 librados: setting wanted keys 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 librados: calling monclient init 2024-09-18T11:54:35.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient: init 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient: _reopen_session rank -1 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient(hunting): picked mon.noname-a con 0x7fe6c806e9d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient(hunting): picked mon.noname-b con 0x7fe6c8135300 addr v1:172.21.15.129:6789/0 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient(hunting): picked mon.noname-c con 0x7fe6c811fb60 addr v1:172.21.15.112:6790/0 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c806ef90 con 0x7fe6c806e9d0 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c809f050 con 0x7fe6c8135300 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c8120c40 con 0x7fe6c811fb60 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient(hunting): _renew_subs 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cfb37700 10 monclient(hunting): authenticate will time out at 1447.306601s 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6cd8d3700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe6c806e9d0 legacy=0x7fe6c81340c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6ce0d4700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe6c811fb60 0x7fe6c814a100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50786/0 (socket says 172.21.15.112:50786) 2024-09-18T11:54:35.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.463+0000 7fe6ce0d4700 1 -- 172.21.15.112:0/889921255 learned_addr learned my addr 172.21.15.112:0/889921255 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6ce0d4700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.112:6790/0 conn(0x7fe6c811fb60 legacy=0x7fe6c814a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 client.?.objecter ms_handle_connect 0x7fe6c811fb60 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117936893 0 0) 0x7fe6c8120c40 con 0x7fe6c811fb60 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): my global_id is 104267 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ac004f10 con 0x7fe6c811fb60 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 client.?.objecter ms_handle_connect 0x7fe6c8135300 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cd0d2700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6789/0 conn(0x7fe6c8135300 legacy=0x7fe6c81469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240057454 0 0) 0x7fe6c809f050 con 0x7fe6c8135300 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): my global_id is 114181 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c8120c40 con 0x7fe6c8135300 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2029082443 0 0) 0x7fe6ac004f10 con 0x7fe6c811fb60 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6c809f050 con 0x7fe6c811fb60 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6c0002ed0 con 0x7fe6c811fb60 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): dump: 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.480 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1618534726 0 0) 0x7fe6c809f050 con 0x7fe6c811fb60 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6789/0 conn(0x7fe6c8135300 legacy=0x7fe6c81469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.112:6789/0 conn(0x7fe6c806e9d0 legacy=0x7fe6c81340c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 monclient: found mon.c 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c8120730 con 0x7fe6c811fb60 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient: _finish_auth 0 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 5 monclient: authenticate success, global_id 104267 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 10 monclient: _renew_subs 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c8142540 con 0x7fe6c811fb60 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 10 client.104267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 10 monclient: _renew_subs 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c8135ba0 con 0x7fe6c811fb60 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 1 librados: init done 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6c0001600 con 0x7fe6c811fb60 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:54:35.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6c0001950 con 0x7fe6c811fb60 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6bbfff700 4 set_mon_vals no callback set 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 10 monclient: dump: 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.483 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe6a8006590 con 0x7fe6a8002840 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.467+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe6c0001ab0 con 0x7fe6c811fb60 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(381..381 src has 1..381) v4 ==== 5409+0+0 (unknown 1825515518 0 0) 0x7fe6c00441d0 con 0x7fe6c811fb60 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 client.104267.objecter ms_dispatch 0x7fe6c8000fd0 osd_map(381..381 src has 1..381) v4 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 3 client.104267.objecter handle_osd_map got epochs [381,381] > 0 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 3 client.104267.objecter handle_osd_map decoding full epoch 381 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 20 client.104267.objecter dump_active .. 0 homeless 2024-09-18T11:54:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6cd0d2700 10 monclient: get_auth_request con 0x7fe6ac0381c0 auth_method 0 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 client.104267.objecter ms_handle_connect 0x7fe6a8002840 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 client.104267.objecter resend_mon_ops 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6cd8d3700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6789/0 conn(0x7fe6a8002840 legacy=0x7fe6a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 client.104267.objecter ms_handle_connect 0x7fe6ac0381c0 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720878958 0 0) 0x7fe6a8006590 con 0x7fe6a8002840 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 monclient(hunting): my global_id is 114184 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c809f050 con 0x7fe6a8002840 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1003612647 0 0) 0x7fe6c809f050 con 0x7fe6a8002840 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6a8006590 con 0x7fe6a8002840 2024-09-18T11:54:35.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6bc001d80 con 0x7fe6a8002840 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.471+0000 7fe6b9ffb700 10 monclient: dump: 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.486 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.475+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4147057352 0 0) 0x7fe6a8006590 con 0x7fe6a8002840 2024-09-18T11:54:35.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.475+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6a800a140 con 0x7fe6a8002840 2024-09-18T11:54:35.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.475+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe6bc001800 con 0x7fe6a8002840 2024-09-18T11:54:35.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.475+0000 7fe6b9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:54:35.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.475+0000 7fe6b9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:54:35.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.475+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6789/0 conn(0x7fe6a8002840 legacy=0x7fe6a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.475+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6789/0 conn(0x7fe6a8002840 legacy=0x7fe6a8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:35.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.483+0000 7fe6cfb37700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:54:35.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.483+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe6a8009670 con 0x7fe6a80090e0 2024-09-18T11:54:35.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6cd8d3700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6789/0 conn(0x7fe6a80090e0 legacy=0x7fe6a8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:54:35.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 10 client.104267.objecter ms_handle_connect 0x7fe6a80090e0 2024-09-18T11:54:35.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 10 client.104267.objecter resend_mon_ops 2024-09-18T11:54:35.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628327806 0 0) 0x7fe6a8009670 con 0x7fe6a80090e0 2024-09-18T11:54:35.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:54:35.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:54:35.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 10 monclient(hunting): my global_id is 114187 2024-09-18T11:54:35.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a8006590 con 0x7fe6a80090e0 2024-09-18T11:54:35.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2427705393 0 0) 0x7fe6a8006590 con 0x7fe6a80090e0 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6a8009670 con 0x7fe6a80090e0 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe6bc004a20 con 0x7fe6a80090e0 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 10 monclient: dump: 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:54:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:54:35.492 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:54:35.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2882471992 0 0) 0x7fe6a8009670 con 0x7fe6a80090e0 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.487+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe6a8009930 con 0x7fe6a80090e0 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4162 (unknown 0 0 989840526) 0x7fe6bc0074d0 con 0x7fe6a80090e0 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6b9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6b9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6789/0 conn(0x7fe6a80090e0 legacy=0x7fe6a8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6b9ffb700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6789/0 conn(0x7fe6a80090e0 legacy=0x7fe6a8008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:35.493 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 48, 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:35.494 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:35.495 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:54:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:54:35.497 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:54:35.498 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:35.499 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:54:35.500 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:54:35.501 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:54:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:35.503 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:54:35.504 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:54:35.505 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:54:35.505 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:54:35.505 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:54:35.505 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:54:35.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:54:35.505 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:54:35.505 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:54:35.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 10 librados: watch_flush enter 2024-09-18T11:54:35.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 10 librados: watch_flush exit 2024-09-18T11:54:35.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 20 client.104267.objecter shutdown clearing up homeless session... 2024-09-18T11:54:35.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 10 client.104267.objecter successfully canceled tick 2024-09-18T11:54:35.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.129:6812/20806 conn(0x7fe6ac0381c0 legacy=0x7fe6ac03a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 10 monclient: shutdown 2024-09-18T11:54:35.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:54:35.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6ce0d4700 1 -- 172.21.15.112:0/889921255 reap_dead start 2024-09-18T11:54:35.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 >> v1:172.21.15.112:6790/0 conn(0x7fe6c811fb60 legacy=0x7fe6c814a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:54:35.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 shutdown_connections 2024-09-18T11:54:35.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 >> 172.21.15.112:0/889921255 conn(0x7fe6c80680d0 msgr2=0x7fe6c809d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:54:35.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 shutdown_connections 2024-09-18T11:54:35.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 1 -- 172.21.15.112:0/889921255 wait complete. 2024-09-18T11:54:35.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:54:35.491+0000 7fe6cfb37700 1 librados: shutdown 2024-09-18T11:54:35.511 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:54:35.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T11:55:01.765 INFO:tasks.workunit.client.0.smithi129.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (17436 ms) 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23052 ms) 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3021 ms) 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3090 ms) 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3026 ms) 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3012 ms) 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (3813 ms) 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38435 ms) 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (40018 ms) 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4052 ms) 2024-09-18T11:55:01.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3033 ms) 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24629 ms) 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14303 ms) 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-18T11:55:01.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4828 ms) 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4041 ms) 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrubs... 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (39623 ms) 2024-09-18T11:55:01.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (30054 ms) 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8971 ms) 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9000 ms) 2024-09-18T11:55:01.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-18T11:55:57.799 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-09-18T11:55:57.795+0000 7f5f70263700 -1 snap_mapper.add_oid found existing snaps mapped on 58:7e6e51f7:test-rados-api-smithi129-23975-97::foo:33, removing 2024-09-18T11:55:57.800 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-18T11:55:57.797+0000 7f51bad4a700 -1 snap_mapper.add_oid found existing snaps mapped on 58:7e6e51f7:test-rados-api-smithi129-23975-97::foo:33, removing 2024-09-18T11:56:05.512 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T11:56:05.512 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T11:56:05.512 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T11:56:05.513 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T11:56:05.515 INFO:tasks.ceph.mon.a:Started 2024-09-18T11:56:05.515 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T11:56:05.516 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T11:56:05.516 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T11:56:05.621 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:05.616+0000 7fbb0c949a00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 412 -> 0 (sync_force? mon rebuild?) 2024-09-18T11:56:20.516 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T11:56:20.517 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T11:56:20.517 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:56:20.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient: get_monmap_and_config 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient: build_initial_monmap 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient: monmap: 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:20.628763+0000 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:20.628763+0000 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.628 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:20.629 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:20.629 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:20.629 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 1 Processor -- start 2024-09-18T11:56:20.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 1 -- start start 2024-09-18T11:56:20.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient: init 2024-09-18T11:56:20.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:20.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient(hunting): picked mon.noname-c con 0x7ffa940a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient(hunting): picked mon.noname-b con 0x7ffa9412edc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient(hunting): picked mon.noname-a con 0x7ffa9412f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa940981a0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9406c120 con 0x7ffa940a2040 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9406eef0 con 0x7ffa9412edc0 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9af3f700 10 monclient(hunting): authenticate will time out at 1552.469707s 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9a73e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ffa9412f8a0 0x7ffa94137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35824/0 (socket says 172.21.15.112:35824) 2024-09-18T11:56:20.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.624+0000 7ffa9a73e700 1 -- 172.21.15.112:0/2114478872 learned_addr learned my addr 172.21.15.112:0/2114478872 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:20.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa99f3d700 1 -- 172.21.15.112:0/2114478872 >> v1:172.21.15.129:6789/0 conn(0x7ffa940a2040 legacy=0x7ffa94097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:20.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9a73e700 1 -- 172.21.15.112:0/2114478872 >> v1:172.21.15.112:6789/0 conn(0x7ffa9412f8a0 legacy=0x7ffa94137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:20.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9973c700 1 -- 172.21.15.112:0/2114478872 >> v1:172.21.15.112:6790/0 conn(0x7ffa9412edc0 legacy=0x7ffa9412f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763975536 0 0) 0x7ffa940981a0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient(hunting): my global_id is 124098 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa80003570 con 0x7ffa9412f8a0 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2601692705 0 0) 0x7ffa80003570 con 0x7ffa9412f8a0 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 >> v1:172.21.15.112:6790/0 conn(0x7ffa9412edc0 legacy=0x7ffa9412f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 >> v1:172.21.15.129:6789/0 conn(0x7ffa940a2040 legacy=0x7ffa94097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: _finish_hunting 0 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 monclient: found mon.noname-a 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:20.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa94096dd0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: _finish_auth 0 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa8c001cb0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9af3f700 5 monclient: authenticate success, global_id 124098 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9af3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: dump: 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:20.635 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9af3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:20.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa8c001e90 con 0x7ffa9412f8a0 2024-09-18T11:56:20.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:20.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9af3f700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:20.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9af3f700 4 set_mon_vals no callback set 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 1 -- 172.21.15.112:0/2114478872 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa8c004c70 con 0x7ffa9412f8a0 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa98f3b700 10 monclient: dump: 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.640 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9af3f700 10 monclient: shutdown 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9af3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.628+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2114478872 >> v1:172.21.15.112:6789/0 conn(0x7ffa9412f8a0 legacy=0x7ffa94137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2114478872 shutdown_connections 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2114478872 >> 172.21.15.112:0/2114478872 conn(0x7ffa94067da0 msgr2=0x7ffa940a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2114478872 shutdown_connections 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2114478872 wait complete. 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient: build_initial_monmap 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient: monmap: 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:20.641 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 librados: starting msgr at 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 librados: starting objecter 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 Processor -- start 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 -- start start 2024-09-18T11:56:20.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 librados: setting wanted keys 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 librados: calling monclient init 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient: init 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient(hunting): picked mon.noname-b con 0x7ffa940a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient(hunting): picked mon.noname-a con 0x7ffa9412edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient(hunting): picked mon.noname-c con 0x7ffa9412f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa94141fc0 con 0x7ffa9412edc0 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9411d7c0 con 0x7ffa940a2040 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9411f2f0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9af3f700 10 monclient(hunting): authenticate will time out at 1552.479407s 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9a73e700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ffa9412f8a0 0x7ffa941498d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37714/0 (socket says 172.21.15.112:37714) 2024-09-18T11:56:20.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.632+0000 7ffa9a73e700 1 -- 172.21.15.112:0/2440197059 learned_addr learned my addr 172.21.15.112:0/2440197059 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:20.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9a73e700 1 -- 172.21.15.112:0/2440197059 >> v1:172.21.15.112:6790/0 conn(0x7ffa9412f8a0 legacy=0x7ffa941498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:20.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 client.?.objecter ms_handle_connect 0x7ffa9412f8a0 2024-09-18T11:56:20.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:20.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423946266 0 0) 0x7ffa9411f2f0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient(hunting): my global_id is 104270 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa78004f10 con 0x7ffa9412f8a0 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa99f3d700 1 -- 172.21.15.112:0/2440197059 >> v1:172.21.15.129:6789/0 conn(0x7ffa940a2040 legacy=0x7ffa94141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 client.?.objecter ms_handle_connect 0x7ffa940a2040 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2985323782 0 0) 0x7ffa78004f10 con 0x7ffa9412f8a0 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa9411f2f0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa8c002000 con 0x7ffa9412f8a0 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient(hunting): dump: 2024-09-18T11:56:20.648 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1883022828 0 0) 0x7ffa9411f2f0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 >> v1:172.21.15.129:6789/0 conn(0x7ffa940a2040 legacy=0x7ffa94141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 >> v1:172.21.15.112:6789/0 conn(0x7ffa9412edc0 legacy=0x7ffa9412e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 monclient: found mon.c 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9411f5d0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient: _finish_auth 0 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 5 monclient: authenticate success, global_id 104270 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 10 monclient: _renew_subs 2024-09-18T11:56:20.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9411f8b0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 10 client.104270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 10 monclient: _renew_subs 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9412e880 con 0x7ffa9412f8a0 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 1 librados: init done 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa8c005210 con 0x7ffa9412f8a0 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa8c005520 con 0x7ffa9412f8a0 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa89ffb700 10 monclient: dump: 2024-09-18T11:56:20.651 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa8bfff700 4 set_mon_vals no callback set 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.636+0000 7ffa9af3f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa9af3f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa9af3f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa740050b0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ffa8c00fda0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(455..455 src has 1..455) v4 ==== 5402+0+0 (unknown 3404193791 0 0) 0x7ffa8c0443c0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa89ffb700 10 client.104270.objecter ms_dispatch 0x7ffa94000fd0 osd_map(455..455 src has 1..455) v4 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa89ffb700 3 client.104270.objecter handle_osd_map got epochs [455,455] > 0 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa89ffb700 3 client.104270.objecter handle_osd_map decoding full epoch 455 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa89ffb700 20 client.104270.objecter dump_active .. 0 homeless 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.640+0000 7ffa99f3d700 10 monclient: get_auth_request con 0x7ffa78038120 auth_method 0 2024-09-18T11:56:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.644+0000 7ffa89ffb700 10 client.104270.objecter ms_handle_connect 0x7ffa78038120 2024-09-18T11:56:20.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.644+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7ffa8c014fe0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.644+0000 7ffa89ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:20.654 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.644+0000 7ffa89ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:20.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.864+0000 7ffa9af3f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:56:20.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.864+0000 7ffa9af3f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:20.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.864+0000 7ffa9af3f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:20.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.864+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffa740058e0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.864+0000 7ffa89ffb700 1 -- 172.21.15.112:0/2440197059 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3370838101) 0x7ffa8c0149e0 con 0x7ffa9412f8a0 2024-09-18T11:56:20.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.864+0000 7ffa89ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:20.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.864+0000 7ffa89ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:20.872 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:56:20.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 10 librados: watch_flush enter 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 10 librados: watch_flush exit 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 20 client.104270.objecter shutdown clearing up homeless session... 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 10 client.104270.objecter successfully canceled tick 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 >> v1:172.21.15.129:6812/20806 conn(0x7ffa78038120 legacy=0x7ffa7803a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 10 monclient: shutdown 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 >> v1:172.21.15.112:6790/0 conn(0x7ffa9412f8a0 legacy=0x7ffa941498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 shutdown_connections 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 >> 172.21.15.112:0/2440197059 conn(0x7ffa94067da0 msgr2=0x7ffa9412fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 shutdown_connections 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 1 -- 172.21.15.112:0/2440197059 wait complete. 2024-09-18T11:56:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.868+0000 7ffa9af3f700 1 librados: shutdown 2024-09-18T11:56:20.886 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:56:20.887 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:56:20.887 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:56:20.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.980+0000 7fa60397b700 10 monclient: get_monmap_and_config 2024-09-18T11:56:20.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.980+0000 7fa60397b700 10 monclient: build_initial_monmap 2024-09-18T11:56:20.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.980+0000 7fa60397b700 10 monclient: monmap: 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:20.986213+0000 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:20.986213+0000 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:20.986 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.980+0000 7fa60397b700 1 Processor -- start 2024-09-18T11:56:20.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.980+0000 7fa60397b700 1 -- start start 2024-09-18T11:56:20.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.980+0000 7fa60397b700 10 monclient: init 2024-09-18T11:56:20.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:20.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:56:20.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 10 monclient(hunting): picked mon.noname-b con 0x7fa5fc0989a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:20.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 10 monclient(hunting): picked mon.noname-c con 0x7fa5fc09d120 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:20.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 10 monclient(hunting): picked mon.noname-a con 0x7fa5fc09d520 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:20.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc098d70 con 0x7fa5fc09d520 2024-09-18T11:56:20.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0981a0 con 0x7fa5fc09d120 2024-09-18T11:56:20.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0972b0 con 0x7fa5fc0989a0 2024-09-18T11:56:20.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:20.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa60397b700 10 monclient(hunting): authenticate will time out at 1552.828891s 2024-09-18T11:56:20.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa601717700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa5fc0989a0 0x7fa5fc09ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37718/0 (socket says 172.21.15.112:37718) 2024-09-18T11:56:20.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa601717700 1 -- 172.21.15.112:0/2101503506 learned_addr learned my addr 172.21.15.112:0/2101503506 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:20.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa601717700 1 -- 172.21.15.112:0/2101503506 >> v1:172.21.15.112:6790/0 conn(0x7fa5fc0989a0 legacy=0x7fa5fc09ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:20.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa601f18700 1 -- 172.21.15.112:0/2101503506 >> v1:172.21.15.112:6789/0 conn(0x7fa5fc09d520 legacy=0x7fa5fc137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:20.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987108934 0 0) 0x7fa5fc0972b0 con 0x7fa5fc0989a0 2024-09-18T11:56:20.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:20.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:20.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 10 monclient(hunting): my global_id is 104273 2024-09-18T11:56:20.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e8004680 con 0x7fa5fc0989a0 2024-09-18T11:56:20.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119564264 0 0) 0x7fa5fc098d70 con 0x7fa5fc09d520 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa600f16700 1 -- 172.21.15.112:0/2101503506 >> v1:172.21.15.129:6789/0 conn(0x7fa5fc09d120 legacy=0x7fa5fc133c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 10 monclient(hunting): my global_id is 124101 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc0972b0 con 0x7fa5fc09d520 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374835999 0 0) 0x7fa5e8004680 con 0x7fa5fc0989a0 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 >> v1:172.21.15.129:6789/0 conn(0x7fa5fc09d120 legacy=0x7fa5fc133c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 >> v1:172.21.15.112:6789/0 conn(0x7fa5fc09d520 legacy=0x7fa5fc137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.984+0000 7fa5f3fff700 10 monclient: _finish_hunting 0 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 1 monclient: found mon.noname-b 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:20.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc09e770 con 0x7fa5fc0989a0 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: _finish_auth 0 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa5f8001940 con 0x7fa5fc0989a0 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 5 monclient: authenticate success, global_id 104273 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: dump: 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5f8001bd0 con 0x7fa5fc0989a0 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 1 -- 172.21.15.112:0/2101503506 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa5f8004a90 con 0x7fa5fc0989a0 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 4 set_mon_vals no callback set 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa5f3fff700 10 monclient: dump: 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient: shutdown 2024-09-18T11:56:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- 172.21.15.112:0/2101503506 >> v1:172.21.15.112:6790/0 conn(0x7fa5fc0989a0 legacy=0x7fa5fc09ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- 172.21.15.112:0/2101503506 shutdown_connections 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- 172.21.15.112:0/2101503506 >> 172.21.15.112:0/2101503506 conn(0x7fa5fc068b30 msgr2=0x7fa5fc0a2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- 172.21.15.112:0/2101503506 shutdown_connections 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- 172.21.15.112:0/2101503506 wait complete. 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient: build_initial_monmap 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient: monmap: 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 librados: starting msgr at 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 librados: starting objecter 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 Processor -- start 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- start start 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 librados: setting wanted keys 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 librados: calling monclient init 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient: init 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient(hunting): picked mon.noname-a con 0x7fa5fc0989a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient(hunting): picked mon.noname-c con 0x7fa5fc09d120 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient(hunting): picked mon.noname-b con 0x7fa5fc09d520 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc09b8f0 con 0x7fa5fc0989a0 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc11ab90 con 0x7fa5fc09d520 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc11ae70 con 0x7fa5fc09d120 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa60397b700 10 monclient(hunting): authenticate will time out at 1552.835480s 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa601717700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa5fc0989a0 0x7fa5fc12da70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35846/0 (socket says 172.21.15.112:35846) 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa600f16700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa5fc09d120 0x7fa5fc142540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37732/0 (socket says 172.21.15.112:37732) 2024-09-18T11:56:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.988+0000 7fa601717700 1 -- 172.21.15.112:0/2505356296 learned_addr learned my addr 172.21.15.112:0/2505356296 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:21.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa601717700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5fc0989a0 legacy=0x7fa5fc12da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 client.?.objecter ms_handle_connect 0x7fa5fc0989a0 2024-09-18T11:56:21.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 client.?.objecter ms_handle_connect 0x7fa5fc09d120 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa600f16700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6790/0 conn(0x7fa5fc09d120 legacy=0x7fa5fc142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213249070 0 0) 0x7fa5fc11ae70 con 0x7fa5fc09d120 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): my global_id is 104276 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e0004f10 con 0x7fa5fc09d120 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531264845 0 0) 0x7fa5fc09b8f0 con 0x7fa5fc0989a0 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): my global_id is 124104 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc11ae70 con 0x7fa5fc0989a0 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1788251894 0 0) 0x7fa5e0004f10 con 0x7fa5fc09d120 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5fc09b8f0 con 0x7fa5fc09d120 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa5ec001c80 con 0x7fa5fc09d120 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): dump: 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428893805 0 0) 0x7fa5fc11ae70 con 0x7fa5fc0989a0 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5e0004f10 con 0x7fa5fc0989a0 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa5f8004830 con 0x7fa5fc0989a0 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient(hunting): dump: 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2891890112 0 0) 0x7fa5fc09b8f0 con 0x7fa5fc09d120 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.129:6789/0 conn(0x7fa5fc09d520 legacy=0x7fa5fc145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5fc0989a0 legacy=0x7fa5fc12da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient: _finish_hunting 0 2024-09-18T11:56:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 monclient: found mon.c 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc119ca0 con 0x7fa5fc09d120 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient: _finish_auth 0 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 5 monclient: authenticate success, global_id 104276 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 10 monclient: _renew_subs 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc119770 con 0x7fa5fc09d120 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 10 client.104276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 10 monclient: _renew_subs 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc12e320 con 0x7fa5fc09d120 2024-09-18T11:56:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa60397b700 1 librados: init done 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5ec002270 con 0x7fa5fc09d120 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa5ec002580 con 0x7fa5fc09d120 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f37fe700 4 set_mon_vals no callback set 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.992+0000 7fa5f17fa700 10 monclient: dump: 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa5ec002750 con 0x7fa5fc09d120 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa60397b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa5dc006590 con 0x7fa5dc002840 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(455..455 src has 1..455) v4 ==== 5402+0+0 (unknown 3404193791 0 0) 0x7fa5ec043e20 con 0x7fa5fc09d120 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 10 client.104276.objecter ms_dispatch 0x7fa5fc000fd0 osd_map(455..455 src has 1..455) v4 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 3 client.104276.objecter handle_osd_map got epochs [455,455] > 0 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 3 client.104276.objecter handle_osd_map decoding full epoch 455 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 20 client.104276.objecter dump_active .. 0 homeless 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa601f18700 10 monclient: get_auth_request con 0x7fa5e0038430 auth_method 0 2024-09-18T11:56:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 10 client.104276.objecter ms_handle_connect 0x7fa5dc002840 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 10 client.104276.objecter resend_mon_ops 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa601717700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc002840 legacy=0x7fa5dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa601717700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc002840 legacy=0x7fa5dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa601717700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc002840 legacy=0x7fa5dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa601717700 1 --1- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc002840 0x7fa5dc004cd0 :-1 s=OPENED pgs=527 cs=1 l=1).handle_message read tag failed 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa601717700 1 --1- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc002840 0x7fa5dc004cd0 :-1 s=OPENED pgs=527 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc002840 legacy=0x7fa5dc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc002840 legacy=0x7fa5dc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:20.996+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa5fc09b8f0 con 0x7fa5e0041b30 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 client.104276.objecter ms_handle_connect 0x7fa5e0038430 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 client.104276.objecter ms_handle_connect 0x7fa5e0041b30 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 client.104276.objecter resend_mon_ops 2024-09-18T11:56:21.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa601717700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5e0041b30 legacy=0x7fa5e0043f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586823667 0 0) 0x7fa5fc09b8f0 con 0x7fa5e0041b30 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 monclient(hunting): my global_id is 124110 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e0004f10 con 0x7fa5e0041b30 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 263130379 0 0) 0x7fa5e0004f10 con 0x7fa5e0041b30 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5fc09b8f0 con 0x7fa5e0041b30 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa5f80051c0 con 0x7fa5e0041b30 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 monclient: dump: 2024-09-18T11:56:21.015 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4036088392 0 0) 0x7fa5fc09b8f0 con 0x7fa5e0041b30 2024-09-18T11:56:21.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5e0046840 con 0x7fa5e0041b30 2024-09-18T11:56:21.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fa5f80073e0 con 0x7fa5e0041b30 2024-09-18T11:56:21.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:21.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5e0041b30 legacy=0x7fa5e0043f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.000+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5e0041b30 legacy=0x7fa5e0043f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:21.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.012+0000 7fa60397b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:56:21.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.012+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa5dc007450 con 0x7fa5dc006820 2024-09-18T11:56:21.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 client.104276.objecter ms_handle_connect 0x7fa5dc006820 2024-09-18T11:56:21.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa601717700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc006820 legacy=0x7fa5dc006c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 client.104276.objecter resend_mon_ops 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964990542 0 0) 0x7fa5dc007450 con 0x7fa5dc006820 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 monclient(hunting): my global_id is 124113 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc09b8f0 con 0x7fa5dc006820 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2450309938 0 0) 0x7fa5fc09b8f0 con 0x7fa5dc006820 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5dc007450 con 0x7fa5dc006820 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa5f80058a0 con 0x7fa5dc006820 2024-09-18T11:56:21.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 monclient: dump: 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.021 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3539641877 0 0) 0x7fa5dc007450 con 0x7fa5dc006820 2024-09-18T11:56:21.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa5dc008760 con 0x7fa5dc006820 2024-09-18T11:56:21.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 3551780176) 0x7fa5f8007690 con 0x7fa5dc006820 2024-09-18T11:56:21.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:56:21.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:21.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc006820 legacy=0x7fa5dc006c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa5f17fa700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6789/0 conn(0x7fa5dc006820 legacy=0x7fa5dc006c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:21.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa601f18700 1 -- 172.21.15.112:0/2505356296 reap_dead start 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 54, 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:56:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:56:21.026 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.027 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.028 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.029 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.030 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.031 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:56:21.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.033 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:56:21.033 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:21.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 10 librados: watch_flush enter 2024-09-18T11:56:21.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 10 librados: watch_flush exit 2024-09-18T11:56:21.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 20 client.104276.objecter shutdown clearing up homeless session... 2024-09-18T11:56:21.034 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 10 client.104276.objecter successfully canceled tick 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.129:6812/20806 conn(0x7fa5e0038430 legacy=0x7fa5e003a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 10 monclient: shutdown 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 >> v1:172.21.15.112:6790/0 conn(0x7fa5fc09d120 legacy=0x7fa5fc142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 shutdown_connections 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 >> 172.21.15.112:0/2505356296 conn(0x7fa5fc068b30 msgr2=0x7fa5fc09ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 shutdown_connections 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 1 -- 172.21.15.112:0/2505356296 wait complete. 2024-09-18T11:56:21.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.016+0000 7fa60397b700 1 librados: shutdown 2024-09-18T11:56:21.037 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:56:21.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:56:21.038 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:56:21.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.140+0000 7f8d3cbf7700 10 monclient: get_monmap_and_config 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.140+0000 7f8d3cbf7700 10 monclient: build_initial_monmap 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.140+0000 7f8d3cbf7700 10 monclient: monmap: 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:21.148336+0000 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:21.148336+0000 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:21.148 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 1 Processor -- start 2024-09-18T11:56:21.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 1 -- start start 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 10 monclient: init 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 10 monclient(hunting): picked mon.noname-b con 0x7f8d3806add0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 10 monclient(hunting): picked mon.noname-a con 0x7f8d38132c30 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 10 monclient(hunting): picked mon.noname-c con 0x7f8d38133070 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d380a17f0 con 0x7f8d38132c30 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d38097010 con 0x7f8d38133070 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d380972f0 con 0x7f8d3806add0 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3cbf7700 10 monclient(hunting): authenticate will time out at 1552.990040s 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8d3806add0 0x7f8d38132520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37744/0 (socket says 172.21.15.112:37744) 2024-09-18T11:56:21.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.144+0000 7f8d3659c700 1 -- 172.21.15.112:0/3470188496 learned_addr learned my addr 172.21.15.112:0/3470188496 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:21.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3659c700 1 -- 172.21.15.112:0/3470188496 >> v1:172.21.15.112:6790/0 conn(0x7f8d3806add0 legacy=0x7f8d38132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d35d9b700 1 -- 172.21.15.112:0/3470188496 >> v1:172.21.15.112:6789/0 conn(0x7f8d38132c30 legacy=0x7f8d38137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d36d9d700 1 -- 172.21.15.112:0/3470188496 >> v1:172.21.15.129:6789/0 conn(0x7f8d38133070 legacy=0x7f8d3813af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226491109 0 0) 0x7f8d380972f0 con 0x7f8d3806add0 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): my global_id is 104279 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d20003570 con 0x7f8d3806add0 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242583806 0 0) 0x7f8d380a17f0 con 0x7f8d38132c30 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): my global_id is 124116 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d380972f0 con 0x7f8d38132c30 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283371057 0 0) 0x7f8d38097010 con 0x7f8d38133070 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient(hunting): my global_id is 114190 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d380a17f0 con 0x7f8d38133070 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4173237927 0 0) 0x7f8d20003570 con 0x7f8d3806add0 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 >> v1:172.21.15.129:6789/0 conn(0x7f8d38133070 legacy=0x7f8d3813af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 >> v1:172.21.15.112:6789/0 conn(0x7f8d38132c30 legacy=0x7f8d38137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: _finish_hunting 0 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 monclient: found mon.noname-b 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d380975d0 con 0x7f8d3806add0 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: _finish_auth 0 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 5 monclient: authenticate success, global_id 104279 2024-09-18T11:56:21.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d28001cc0 con 0x7f8d3806add0 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: dump: 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d28001f50 con 0x7f8d3806add0 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 1 -- 172.21.15.112:0/3470188496 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d28004e10 con 0x7f8d3806add0 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 4 set_mon_vals no callback set 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3559a700 10 monclient: dump: 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.157 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 10 monclient: shutdown 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3470188496 >> v1:172.21.15.112:6790/0 conn(0x7f8d3806add0 legacy=0x7f8d38132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3470188496 shutdown_connections 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3470188496 >> 172.21.15.112:0/3470188496 conn(0x7f8d380a5f90 msgr2=0x7f8d380a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3470188496 shutdown_connections 2024-09-18T11:56:21.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3470188496 wait complete. 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 10 monclient: build_initial_monmap 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 10 monclient: monmap: 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:21.164 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 librados: starting msgr at 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 librados: starting objecter 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 Processor -- start 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 -- start start 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 librados: setting wanted keys 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 1 librados: calling monclient init 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.148+0000 7f8d3cbf7700 10 monclient: init 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 10 monclient(hunting): picked mon.noname-c con 0x7f8d3806add0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 10 monclient(hunting): picked mon.noname-a con 0x7f8d38132c30 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:21.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 10 monclient(hunting): picked mon.noname-b con 0x7f8d38133070 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3811ec40 con 0x7f8d38132c30 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3811e960 con 0x7f8d38133070 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3809e1e0 con 0x7f8d3806add0 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3cbf7700 10 monclient(hunting): authenticate will time out at 1552.996815s 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d35d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8d38132c30 0x7f8d38146bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35886/0 (socket says 172.21.15.112:35886) 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d35d9b700 1 -- 172.21.15.112:0/3334473456 learned_addr learned my addr 172.21.15.112:0/3334473456 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d35d9b700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6789/0 conn(0x7f8d38132c30 legacy=0x7f8d38146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 client.?.objecter ms_handle_connect 0x7f8d38132c30 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d3659c700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6790/0 conn(0x7f8d3806add0 legacy=0x7f8d38131560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 client.?.objecter ms_handle_connect 0x7f8d3806add0 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788083233 0 0) 0x7f8d3811ec40 con 0x7f8d38132c30 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): my global_id is 124119 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d18004f10 con 0x7f8d38132c30 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079678694 0 0) 0x7f8d3809e1e0 con 0x7f8d3806add0 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): my global_id is 104282 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d36d9d700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.129:6789/0 conn(0x7f8d38133070 legacy=0x7f8d3814a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3811ec40 con 0x7f8d3806add0 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 client.?.objecter ms_handle_connect 0x7f8d38133070 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130663569 0 0) 0x7f8d3811e960 con 0x7f8d38133070 2024-09-18T11:56:21.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 10 monclient(hunting): my global_id is 114193 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3809e1e0 con 0x7f8d38133070 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748222990 0 0) 0x7f8d3811ec40 con 0x7f8d3806add0 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.152+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3811e960 con 0x7f8d3806add0 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2305052407 0 0) 0x7f8d18004f10 con 0x7f8d38132c30 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3811ec40 con 0x7f8d38132c30 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d2c0019c0 con 0x7f8d38132c30 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient(hunting): dump: 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.168 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d28004bb0 con 0x7f8d3806add0 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient(hunting): dump: 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.169 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 640424844 0 0) 0x7f8d3811e960 con 0x7f8d3806add0 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.129:6789/0 conn(0x7f8d38133070 legacy=0x7f8d3814a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6789/0 conn(0x7f8d38132c30 legacy=0x7f8d38146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient: _finish_hunting 0 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 monclient: found mon.c 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:21.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3811e680 con 0x7f8d3806add0 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient: _finish_auth 0 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 5 monclient: authenticate success, global_id 104282 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 10 monclient: _renew_subs 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d38141d70 con 0x7f8d3806add0 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 10 client.104282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 10 monclient: _renew_subs 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d38131c70 con 0x7f8d3806add0 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 1 librados: init done 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d280016a0 con 0x7f8d3806add0 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:21.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d280053c0 con 0x7f8d3806add0 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 monclient: dump: 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d34d99700 4 set_mon_vals no callback set 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8d1c006590 con 0x7f8d1c002840 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8d2800fc40 con 0x7f8d3806add0 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(455..455 src has 1..455) v4 ==== 5402+0+0 (unknown 3404193791 0 0) 0x7f8d28044210 con 0x7f8d3806add0 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 10 client.104282.objecter ms_dispatch 0x7f8d38000fd0 osd_map(455..455 src has 1..455) v4 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 3 client.104282.objecter handle_osd_map got epochs [455,455] > 0 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 3 client.104282.objecter handle_osd_map decoding full epoch 455 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d267fc700 20 client.104282.objecter dump_active .. 0 homeless 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.156+0000 7f8d36d9d700 10 monclient: get_auth_request con 0x7f8d180383d0 auth_method 0 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 client.104282.objecter ms_handle_connect 0x7f8d1c002840 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 client.104282.objecter resend_mon_ops 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d35d9b700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6790/0 conn(0x7f8d1c002840 legacy=0x7f8d1c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606668339 0 0) 0x7f8d1c006590 con 0x7f8d1c002840 2024-09-18T11:56:21.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 monclient(hunting): my global_id is 104285 2024-09-18T11:56:21.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3811e960 con 0x7f8d1c002840 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3921608224 0 0) 0x7f8d3811e960 con 0x7f8d1c002840 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d1c006590 con 0x7f8d1c002840 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d2c0027e0 con 0x7f8d1c002840 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 monclient: dump: 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.175 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2789199949 0 0) 0x7f8d1c006590 con 0x7f8d1c002840 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d1c00a140 con 0x7f8d1c002840 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 client.104282.objecter ms_handle_connect 0x7f8d180383d0 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f8d2c002a70 con 0x7f8d1c002840 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6790/0 conn(0x7f8d1c002840 legacy=0x7f8d1c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.160+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6790/0 conn(0x7f8d1c002840 legacy=0x7f8d1c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:21.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.172+0000 7f8d3cbf7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:56:21.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.172+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8d1c007530 con 0x7f8d1c0068e0 2024-09-18T11:56:21.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d35d9b700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6790/0 conn(0x7f8d1c0068e0 legacy=0x7f8d1c006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 client.104282.objecter ms_handle_connect 0x7f8d1c0068e0 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 client.104282.objecter resend_mon_ops 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207928593 0 0) 0x7f8d1c007530 con 0x7f8d1c0068e0 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 monclient(hunting): my global_id is 104288 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1c006590 con 0x7f8d1c0068e0 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4213659274 0 0) 0x7f8d1c006590 con 0x7f8d1c0068e0 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d1c007530 con 0x7f8d1c0068e0 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8d2c0059b0 con 0x7f8d1c0068e0 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 monclient: dump: 2024-09-18T11:56:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1882326341 0 0) 0x7f8d1c007530 con 0x7f8d1c0068e0 2024-09-18T11:56:21.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8d1c0087b0 con 0x7f8d1c0068e0 2024-09-18T11:56:21.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 3234963044) 0x7f8d2c005d80 con 0x7f8d1c0068e0 2024-09-18T11:56:21.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:56:21.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:21.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6790/0 conn(0x7f8d1c0068e0 legacy=0x7f8d1c006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d267fc700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6790/0 conn(0x7f8d1c0068e0 legacy=0x7f8d1c006d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:21.183 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 54, 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:56:21.184 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.185 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:56:21.186 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.187 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.188 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.189 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.191 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.192 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.192 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:56:21.192 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.192 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.192 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:21.193 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.194 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.195 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.195 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:56:21.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.195 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.195 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:56:21.195 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:21.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 10 librados: watch_flush enter 2024-09-18T11:56:21.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 10 librados: watch_flush exit 2024-09-18T11:56:21.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 20 client.104282.objecter shutdown clearing up homeless session... 2024-09-18T11:56:21.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 10 client.104282.objecter successfully canceled tick 2024-09-18T11:56:21.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.129:6812/20806 conn(0x7f8d180383d0 legacy=0x7f8d1803a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 10 monclient: shutdown 2024-09-18T11:56:21.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:21.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 >> v1:172.21.15.112:6790/0 conn(0x7f8d3806add0 legacy=0x7f8d38131560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d36d9d700 1 -- 172.21.15.112:0/3334473456 reap_dead start 2024-09-18T11:56:21.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d36d9d700 1 -- 172.21.15.112:0/3334473456 reap_dead start 2024-09-18T11:56:21.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 shutdown_connections 2024-09-18T11:56:21.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 >> 172.21.15.112:0/3334473456 conn(0x7f8d380a5f90 msgr2=0x7f8d380a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:21.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 shutdown_connections 2024-09-18T11:56:21.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 1 -- 172.21.15.112:0/3334473456 wait complete. 2024-09-18T11:56:21.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.176+0000 7f8d3cbf7700 1 librados: shutdown 2024-09-18T11:56:21.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:56:21.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:56:21.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient: get_monmap_and_config 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient: build_initial_monmap 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient: monmap: 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:21.312671+0000 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:21.312671+0000 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:21.312 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.313 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:21.313 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:21.313 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:21.313 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 1 Processor -- start 2024-09-18T11:56:21.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 1 -- start start 2024-09-18T11:56:21.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient: init 2024-09-18T11:56:21.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:21.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:21.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient(hunting): picked mon.noname-b con 0x7f0f300a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:21.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient(hunting): picked mon.noname-a con 0x7f0f3012edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:21.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient(hunting): picked mon.noname-c con 0x7f0f3012f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:21.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f300981a0 con 0x7f0f3012edc0 2024-09-18T11:56:21.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3006c120 con 0x7f0f3012f8a0 2024-09-18T11:56:21.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3006eef0 con 0x7f0f300a2040 2024-09-18T11:56:21.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:21.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f36977700 10 monclient(hunting): authenticate will time out at 1553.153867s 2024-09-18T11:56:21.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f35174700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0f3012edc0 0x7f0f3012f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35896/0 (socket says 172.21.15.112:35896) 2024-09-18T11:56:21.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f35174700 1 -- 172.21.15.112:0/449682023 learned_addr learned my addr 172.21.15.112:0/449682023 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:21.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.308+0000 7f0f35975700 1 --1- 172.21.15.112:0/449682023 >> v1:172.21.15.112:6790/0 conn(0x7f0f300a2040 0x7f0f30097a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37784/0 (socket says 172.21.15.112:37784) 2024-09-18T11:56:21.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f35174700 1 -- 172.21.15.112:0/449682023 >> v1:172.21.15.112:6789/0 conn(0x7f0f3012edc0 legacy=0x7f0f3012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f35975700 1 -- 172.21.15.112:0/449682023 >> v1:172.21.15.112:6790/0 conn(0x7f0f300a2040 legacy=0x7f0f30097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285612443 0 0) 0x7f0f3006eef0 con 0x7f0f300a2040 2024-09-18T11:56:21.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): my global_id is 104291 2024-09-18T11:56:21.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1c003570 con 0x7f0f300a2040 2024-09-18T11:56:21.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36176700 1 -- 172.21.15.112:0/449682023 >> v1:172.21.15.129:6789/0 conn(0x7f0f3012f8a0 legacy=0x7f0f30137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414245578 0 0) 0x7f0f300981a0 con 0x7f0f3012edc0 2024-09-18T11:56:21.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): my global_id is 124122 2024-09-18T11:56:21.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3006eef0 con 0x7f0f3012edc0 2024-09-18T11:56:21.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308826602 0 0) 0x7f0f3006c120 con 0x7f0f3012f8a0 2024-09-18T11:56:21.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient(hunting): my global_id is 114196 2024-09-18T11:56:21.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f300981a0 con 0x7f0f3012f8a0 2024-09-18T11:56:21.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3977379491 0 0) 0x7f0f1c003570 con 0x7f0f300a2040 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 >> v1:172.21.15.129:6789/0 conn(0x7f0f3012f8a0 legacy=0x7f0f30137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 >> v1:172.21.15.112:6789/0 conn(0x7f0f3012edc0 legacy=0x7f0f3012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: _finish_hunting 0 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 monclient: found mon.noname-b 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f30096dd0 con 0x7f0f300a2040 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: _finish_auth 0 2024-09-18T11:56:21.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0f24001940 con 0x7f0f300a2040 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 5 monclient: authenticate success, global_id 104291 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: dump: 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.326 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.327 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.329 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.329 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.330 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.330 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:21.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f24001bd0 con 0x7f0f300a2040 2024-09-18T11:56:21.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:21.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 1 -- 172.21.15.112:0/449682023 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0f240049d0 con 0x7f0f300a2040 2024-09-18T11:56:21.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:21.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 4 set_mon_vals no callback set 2024-09-18T11:56:21.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:21.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f34973700 10 monclient: dump: 2024-09-18T11:56:21.332 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.332 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.332 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.332 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.332 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 10 monclient: shutdown 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 1 -- 172.21.15.112:0/449682023 >> v1:172.21.15.112:6790/0 conn(0x7f0f300a2040 legacy=0x7f0f30097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 1 -- 172.21.15.112:0/449682023 shutdown_connections 2024-09-18T11:56:21.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 1 -- 172.21.15.112:0/449682023 >> 172.21.15.112:0/449682023 conn(0x7f0f30067da0 msgr2=0x7f0f300a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:21.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 1 -- 172.21.15.112:0/449682023 shutdown_connections 2024-09-18T11:56:21.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 1 -- 172.21.15.112:0/449682023 wait complete. 2024-09-18T11:56:21.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 10 monclient: build_initial_monmap 2024-09-18T11:56:21.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.312+0000 7f0f36977700 10 monclient: monmap: 2024-09-18T11:56:21.334 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:21.334 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:21.334 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:21.335 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:21.335 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:21.335 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.335 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:21.335 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:21.335 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:21.335 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 librados: starting msgr at 2024-09-18T11:56:21.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 librados: starting objecter 2024-09-18T11:56:21.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 Processor -- start 2024-09-18T11:56:21.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 -- start start 2024-09-18T11:56:21.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 librados: setting wanted keys 2024-09-18T11:56:21.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 librados: calling monclient init 2024-09-18T11:56:21.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 10 monclient: init 2024-09-18T11:56:21.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 10 monclient(hunting): picked mon.noname-a con 0x7f0f300a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 10 monclient(hunting): picked mon.noname-b con 0x7f0f3012edc0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 10 monclient(hunting): picked mon.noname-c con 0x7f0f3012f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f300a02d0 con 0x7f0f300a2040 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3013e830 con 0x7f0f3012edc0 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3013efe0 con 0x7f0f3012f8a0 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36977700 10 monclient(hunting): authenticate will time out at 1553.162007s 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36176700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f0f3012f8a0 0x7f0f30146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37798/0 (socket says 172.21.15.112:37798) 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.316+0000 7f0f36176700 1 -- 172.21.15.112:0/2878911023 learned_addr learned my addr 172.21.15.112:0/2878911023 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36176700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.112:6790/0 conn(0x7f0f3012f8a0 legacy=0x7f0f30146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 client.?.objecter ms_handle_connect 0x7f0f3012f8a0 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f35975700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.112:6789/0 conn(0x7f0f300a2040 legacy=0x7f0f3013e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 client.?.objecter ms_handle_connect 0x7f0f300a2040 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 client.?.objecter ms_handle_connect 0x7f0f3012edc0 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f35174700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6789/0 conn(0x7f0f3012edc0 legacy=0x7f0f30137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043127369 0 0) 0x7f0f3013efe0 con 0x7f0f3012f8a0 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): my global_id is 104294 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f14004f10 con 0x7f0f3012f8a0 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904884248 0 0) 0x7f0f300a02d0 con 0x7f0f300a2040 2024-09-18T11:56:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): my global_id is 124125 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3013efe0 con 0x7f0f300a2040 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657945474 0 0) 0x7f0f3013e830 con 0x7f0f3012edc0 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): my global_id is 114199 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f300a02d0 con 0x7f0f3012edc0 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 692971390 0 0) 0x7f0f3013efe0 con 0x7f0f300a2040 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f3013e830 con 0x7f0f300a2040 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0f24004830 con 0x7f0f300a2040 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:21.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): dump: 2024-09-18T11:56:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.348 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.349 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 341997054 0 0) 0x7f0f14004f10 con 0x7f0f3012f8a0 2024-09-18T11:56:21.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f3013efe0 con 0x7f0f3012f8a0 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0f2c001cd0 con 0x7f0f3012f8a0 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient(hunting): dump: 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.350 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4129376076 0 0) 0x7f0f3013e830 con 0x7f0f300a2040 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.112:6790/0 conn(0x7f0f3012f8a0 legacy=0x7f0f30146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6789/0 conn(0x7f0f3012edc0 legacy=0x7f0f30137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 monclient: found mon.a 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f301346a0 con 0x7f0f300a2040 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient: _finish_auth 0 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 5 monclient: authenticate success, global_id 124125 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 10 monclient: _renew_subs 2024-09-18T11:56:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f30135590 con 0x7f0f300a2040 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 10 client.124125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 10 monclient: _renew_subs 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f30137a70 con 0x7f0f300a2040 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 1 librados: init done 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f24004a60 con 0x7f0f300a2040 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0f240051a0 con 0x7f0f300a2040 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 monclient: dump: 2024-09-18T11:56:21.352 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f237fe700 4 set_mon_vals no callback set 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0f10006590 con 0x7f0f10002840 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0f2400faa0 con 0x7f0f300a2040 2024-09-18T11:56:21.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(455..455 src has 1..455) v4 ==== 5402+0+0 (unknown 3404193791 0 0) 0x7f0f24043fc0 con 0x7f0f300a2040 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 10 client.124125.objecter ms_dispatch 0x7f0f30000fd0 osd_map(455..455 src has 1..455) v4 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 3 client.124125.objecter handle_osd_map got epochs [455,455] > 0 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.320+0000 7f0f21ffb700 3 client.124125.objecter handle_osd_map decoding full epoch 455 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 20 client.124125.objecter dump_active .. 0 homeless 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f36176700 10 monclient: get_auth_request con 0x7f0f1403a460 auth_method 0 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f35174700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6789/0 conn(0x7f0f10002840 legacy=0x7f0f10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 client.124125.objecter ms_handle_connect 0x7f0f10002840 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 client.124125.objecter resend_mon_ops 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863877204 0 0) 0x7f0f10006590 con 0x7f0f10002840 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 monclient(hunting): my global_id is 114202 2024-09-18T11:56:21.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3013e830 con 0x7f0f10002840 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 client.124125.objecter ms_handle_connect 0x7f0f1403a460 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3254396701 0 0) 0x7f0f3013e830 con 0x7f0f10002840 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f10006590 con 0x7f0f10002840 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0f28004f40 con 0x7f0f10002840 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 10 monclient: dump: 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.355 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2178454200 0 0) 0x7f0f10006590 con 0x7f0f10002840 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.324+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f1000a140 con 0x7f0f10002840 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.328+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f0f28007100 con 0x7f0f10002840 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.328+0000 7f0f21ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.328+0000 7f0f21ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:21.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.328+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6789/0 conn(0x7f0f10002840 legacy=0x7f0f10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.328+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6789/0 conn(0x7f0f10002840 legacy=0x7f0f10004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:21.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.336+0000 7f0f36977700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:56:21.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.336+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0f10007550 con 0x7f0f10006900 2024-09-18T11:56:21.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f35174700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6789/0 conn(0x7f0f10006900 legacy=0x7f0f10006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 10 client.124125.objecter ms_handle_connect 0x7f0f10006900 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 10 client.124125.objecter resend_mon_ops 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079510451 0 0) 0x7f0f10007550 con 0x7f0f10006900 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 10 monclient(hunting): my global_id is 114205 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f10006590 con 0x7f0f10006900 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2282279962 0 0) 0x7f0f10006590 con 0x7f0f10006900 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f10007550 con 0x7f0f10006900 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0f28005620 con 0x7f0f10006900 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:21.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 10 monclient: dump: 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 913296685 0 0) 0x7f0f10007550 con 0x7f0f10006900 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.340+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0f100087b0 con 0x7f0f10006900 2024-09-18T11:56:21.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 1495314480) 0x7f0f28006a60 con 0x7f0f10006900 2024-09-18T11:56:21.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f21ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:56:21.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f21ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:21.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6789/0 conn(0x7f0f10006900 legacy=0x7f0f10006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f21ffb700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6789/0 conn(0x7f0f10006900 legacy=0x7f0f10006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:21.361 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:21.361 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:21.361 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 54, 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.362 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.363 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:56:21.364 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:21.365 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:21.373 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:56:21.373 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:21.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 10 librados: watch_flush enter 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 10 librados: watch_flush exit 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 20 client.124125.objecter shutdown clearing up homeless session... 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 10 client.124125.objecter successfully canceled tick 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.129:6812/20806 conn(0x7f0f1403a460 legacy=0x7f0f1403c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 10 monclient: shutdown 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 >> v1:172.21.15.112:6789/0 conn(0x7f0f300a2040 legacy=0x7f0f3013e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36176700 1 -- 172.21.15.112:0/2878911023 reap_dead start 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36176700 1 -- 172.21.15.112:0/2878911023 reap_dead start 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 shutdown_connections 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 >> 172.21.15.112:0/2878911023 conn(0x7f0f30067da0 msgr2=0x7f0f3006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 shutdown_connections 2024-09-18T11:56:21.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 1 -- 172.21.15.112:0/2878911023 wait complete. 2024-09-18T11:56:21.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.344+0000 7f0f36977700 1 librados: shutdown 2024-09-18T11:56:21.376 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:56:21.376 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T11:56:21.376 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T11:56:21.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.484+0000 7f5cb0a72700 10 monclient: get_monmap_and_config 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.484+0000 7f5cb0a72700 10 monclient: build_initial_monmap 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.484+0000 7f5cb0a72700 10 monclient: monmap: 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:21.491496+0000 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:21.491496+0000 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:21.491 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:21.492 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:21.492 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.484+0000 7f5cb0a72700 1 Processor -- start 2024-09-18T11:56:21.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.484+0000 7f5cb0a72700 1 -- start start 2024-09-18T11:56:21.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.484+0000 7f5cb0a72700 10 monclient: init 2024-09-18T11:56:21.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:21.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:21.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 10 monclient(hunting): picked mon.noname-c con 0x7f5cac0a2040 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:21.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 10 monclient(hunting): picked mon.noname-b con 0x7f5cac12edc0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:21.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 10 monclient(hunting): picked mon.noname-a con 0x7f5cac12f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:21.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac0981a0 con 0x7f5cac12f8a0 2024-09-18T11:56:21.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac06c120 con 0x7f5cac0a2040 2024-09-18T11:56:21.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac06eef0 con 0x7f5cac12edc0 2024-09-18T11:56:21.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:21.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 10 monclient(hunting): authenticate will time out at 1553.332369s 2024-09-18T11:56:21.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cabfff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5cac12f8a0 0x7f5cac137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35902/0 (socket says 172.21.15.112:35902) 2024-09-18T11:56:21.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cabfff700 1 -- 172.21.15.112:0/2316444614 learned_addr learned my addr 172.21.15.112:0/2316444614 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cabfff700 1 -- 172.21.15.112:0/2316444614 >> v1:172.21.15.112:6789/0 conn(0x7f5cac12f8a0 legacy=0x7f5cac137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5ca3fff700 1 -- 172.21.15.112:0/2316444614 >> v1:172.21.15.112:6790/0 conn(0x7f5cac12edc0 legacy=0x7f5cac12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cab7fe700 1 -- 172.21.15.112:0/2316444614 >> v1:172.21.15.129:6789/0 conn(0x7f5cac0a2040 legacy=0x7f5cac097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684175114 0 0) 0x7f5cac0981a0 con 0x7f5cac12f8a0 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): my global_id is 124128 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c98003570 con 0x7f5cac12f8a0 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351351369 0 0) 0x7f5cac06eef0 con 0x7f5cac12edc0 2024-09-18T11:56:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): my global_id is 104297 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cac0981a0 con 0x7f5cac12edc0 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790207425 0 0) 0x7f5cac06c120 con 0x7f5cac0a2040 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient(hunting): my global_id is 114208 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cac06eef0 con 0x7f5cac0a2040 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702218168 0 0) 0x7f5c98003570 con 0x7f5cac12f8a0 2024-09-18T11:56:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 >> v1:172.21.15.112:6790/0 conn(0x7f5cac12edc0 legacy=0x7f5cac12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 >> v1:172.21.15.129:6789/0 conn(0x7f5cac0a2040 legacy=0x7f5cac097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient: _finish_hunting 0 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 monclient: found mon.noname-a 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac096dd0 con 0x7f5cac12f8a0 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient: _finish_auth 0 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5ca4001d80 con 0x7f5cac12f8a0 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 5 monclient: authenticate success, global_id 124128 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:21.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5caaffd700 10 monclient: dump: 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.488+0000 7f5cb0a72700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ca4001fe0 con 0x7f5cac12f8a0 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5caaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 4 set_mon_vals no callback set 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5caaffd700 1 -- 172.21.15.112:0/2316444614 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5ca4004e10 con 0x7f5cac12f8a0 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5caaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5caaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5caaffd700 10 monclient: dump: 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient: shutdown 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2316444614 >> v1:172.21.15.112:6789/0 conn(0x7f5cac12f8a0 legacy=0x7f5cac137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2316444614 shutdown_connections 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2316444614 >> 172.21.15.112:0/2316444614 conn(0x7f5cac067da0 msgr2=0x7f5cac0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2316444614 shutdown_connections 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2316444614 wait complete. 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient: build_initial_monmap 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient: monmap: 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:21.503 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 librados: starting msgr at 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 librados: starting objecter 2024-09-18T11:56:21.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 Processor -- start 2024-09-18T11:56:21.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- start start 2024-09-18T11:56:21.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 librados: setting wanted keys 2024-09-18T11:56:21.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 librados: calling monclient init 2024-09-18T11:56:21.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient: init 2024-09-18T11:56:21.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:21.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:21.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient(hunting): picked mon.noname-c con 0x7f5cac0a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient(hunting): picked mon.noname-a con 0x7f5cac12edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient(hunting): picked mon.noname-b con 0x7f5cac12f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac06f110 con 0x7f5cac12edc0 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac09f8c0 con 0x7f5cac12f8a0 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac1199b0 con 0x7f5cac0a2040 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5ca3fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5cac12edc0 0x7f5cac1422d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35918/0 (socket says 172.21.15.112:35918) 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cb0a72700 10 monclient(hunting): authenticate will time out at 1553.339964s 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5ca3fff700 1 -- 172.21.15.112:0/2957723417 learned_addr learned my addr 172.21.15.112:0/2957723417 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.492+0000 7f5cab7fe700 1 --1- 172.21.15.112:0/2957723417 >> v1:172.21.15.112:6790/0 conn(0x7f5cac0a2040 0x7f5cac12de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37824/0 (socket says 172.21.15.112:37824) 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca3fff700 1 -- 172.21.15.112:0/2957723417 >> v1:172.21.15.112:6789/0 conn(0x7f5cac12edc0 legacy=0x7f5cac1422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 client.?.objecter ms_handle_connect 0x7f5cac12edc0 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 client.?.objecter ms_handle_connect 0x7f5cac0a2040 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cab7fe700 1 -- 172.21.15.112:0/2957723417 >> v1:172.21.15.112:6790/0 conn(0x7f5cac0a2040 legacy=0x7f5cac12de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096444529 0 0) 0x7f5cac06f110 con 0x7f5cac12edc0 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): my global_id is 124131 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c004f10 con 0x7f5cac12edc0 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cab7fe700 1 --1- 172.21.15.112:0/2957723417 >> v1:172.21.15.112:6790/0 conn(0x7f5cac0a2040 0x7f5cac12de40 :-1 s=OPENED pgs=9108 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f5cac1199b0 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 client.?.objecter ms_handle_connect 0x7f5cac12f8a0 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cabfff700 1 -- 172.21.15.112:0/2957723417 >> v1:172.21.15.129:6789/0 conn(0x7f5cac12f8a0 legacy=0x7f5cac145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:21.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777010789 0 0) 0x7f5cac09f8c0 con 0x7f5cac12f8a0 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): my global_id is 114211 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cac06f110 con 0x7f5cac12f8a0 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2722570476 0 0) 0x7f5c8c004f10 con 0x7f5cac12edc0 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cac09f8c0 con 0x7f5cac12edc0 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c9c001c60 con 0x7f5cac12edc0 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): dump: 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.511 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4279767548 0 0) 0x7f5cac09f8c0 con 0x7f5cac12edc0 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 >> v1:172.21.15.112:6790/0 conn(0x7f5cac0a2040 legacy=0x7f5cac12de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 >> v1:172.21.15.129:6789/0 conn(0x7f5cac12f8a0 legacy=0x7f5cac145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient: _finish_hunting 0 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 monclient: found mon.a 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac1194a0 con 0x7f5cac12edc0 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient: _finish_auth 0 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 5 monclient: authenticate success, global_id 124131 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51794275 0 0) 0x7f5cac1199b0 con 0x7f5cac0a2040 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 10 monclient: _renew_subs 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cac11a070 con 0x7f5cac12edc0 2024-09-18T11:56:21.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 10 client.124131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 10 monclient: _renew_subs 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c9c002300 con 0x7f5cac12edc0 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cac12e550 con 0x7f5cac12edc0 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5cb0a72700 1 librados: init done 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5c9c002610 con 0x7f5cac12edc0 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5caa7fc700 4 set_mon_vals no callback set 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.496+0000 7f5ca37fe700 10 monclient: dump: 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:21.514 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5cb0a72700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5cb0a72700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5cb0a72700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cac05f2e0 con 0x7f5cac12edc0 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5c9c002770 con 0x7f5cac12edc0 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(455..455 src has 1..455) v4 ==== 5402+0+0 (unknown 3404193791 0 0) 0x7f5c9c043e90 con 0x7f5cac12edc0 2024-09-18T11:56:21.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5ca37fe700 10 client.124131.objecter ms_dispatch 0x7f5cac000fd0 osd_map(455..455 src has 1..455) v4 2024-09-18T11:56:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5ca37fe700 3 client.124131.objecter handle_osd_map got epochs [455,455] > 0 2024-09-18T11:56:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5ca37fe700 3 client.124131.objecter handle_osd_map decoding full epoch 455 2024-09-18T11:56:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5ca37fe700 20 client.124131.objecter dump_active .. 0 homeless 2024-09-18T11:56:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.500+0000 7f5cab7fe700 10 monclient: get_auth_request con 0x7f5c8c0381b0 auth_method 0 2024-09-18T11:56:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.504+0000 7f5ca37fe700 10 client.124131.objecter ms_handle_connect 0x7f5c8c0381b0 2024-09-18T11:56:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.508+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f5c9c014a50 con 0x7f5cac12edc0 2024-09-18T11:56:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.508+0000 7f5ca37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.508+0000 7f5ca37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:21.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.736+0000 7f5cb0a72700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T11:56:21.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.736+0000 7f5cb0a72700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T11:56:21.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.736+0000 7f5cb0a72700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:21.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.736+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f5cac063680 con 0x7f5cac12edc0 2024-09-18T11:56:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.736+0000 7f5ca37fe700 1 -- 172.21.15.112:0/2957723417 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f5c9c014450 con 0x7f5cac12edc0 2024-09-18T11:56:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.736+0000 7f5ca37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T11:56:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.736+0000 7f5ca37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:21.743 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.736+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.743 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.736+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=1,health=11,kv=8,logm=60} crc {auth=3432601492,config=1228700967,health=77634962,kv=1310296871,logm=2320259675}) 2024-09-18T11:56:21.743 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.736+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=799965914,config=3140514009,health=418924436,kv=1310296871,logm=725845139}) 2024-09-18T11:56:21.743 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.736+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.743 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.736+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=1,health=11,kv=8,logm=60} crc {auth=3432601492,config=1228700967,health=77634962,kv=1310296871,logm=2320259675}) 2024-09-18T11:56:21.743 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.736+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=1,health=12,kv=8,logm=60} crc {auth=987705972,config=1228700967,health=418924436,kv=1310296871,logm=2611749319}) 2024-09-18T11:56:21.745 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.740+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.745 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.740+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=53436686}) 2024-09-18T11:56:21.745 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.740+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312774513}) 2024-09-18T11:56:21.746 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.740+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.746 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.740+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=53436686}) 2024-09-18T11:56:21.746 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.740+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703797422}) 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 10 librados: watch_flush enter 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 10 librados: watch_flush exit 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 20 client.124131.objecter shutdown clearing up homeless session... 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 10 client.124131.objecter successfully canceled tick 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 >> v1:172.21.15.129:6812/20806 conn(0x7f5c8c0381b0 legacy=0x7f5c8c03a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 10 monclient: shutdown 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 >> v1:172.21.15.112:6789/0 conn(0x7f5cac12edc0 legacy=0x7f5cac1422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:21.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 shutdown_connections 2024-09-18T11:56:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 >> 172.21.15.112:0/2957723417 conn(0x7f5cac067da0 msgr2=0x7f5cac06c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 shutdown_connections 2024-09-18T11:56:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 1 -- 172.21.15.112:0/2957723417 wait complete. 2024-09-18T11:56:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:21.740+0000 7f5cb0a72700 1 librados: shutdown 2024-09-18T11:56:21.748 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392707877}) 2024-09-18T11:56:21.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3003804721}) 2024-09-18T11:56:21.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392707877}) 2024-09-18T11:56:21.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100132506}) 2024-09-18T11:56:21.750 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.751 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485891264}) 2024-09-18T11:56:21.751 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553103344}) 2024-09-18T11:56:21.751 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.751 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485891264}) 2024-09-18T11:56:21.751 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.744+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3785010913}) 2024-09-18T11:56:21.753 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.753 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866751655}) 2024-09-18T11:56:21.754 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339036897}) 2024-09-18T11:56:21.754 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.754 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866751655}) 2024-09-18T11:56:21.754 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407032294}) 2024-09-18T11:56:21.755 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.755 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773482841}) 2024-09-18T11:56:21.755 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3269208585}) 2024-09-18T11:56:21.756 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.756 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773482841}) 2024-09-18T11:56:21.756 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.748+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939047769}) 2024-09-18T11:56:21.757 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.757 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32430288}) 2024-09-18T11:56:21.757 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4059327478}) 2024-09-18T11:56:21.757 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.757 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32430288}) 2024-09-18T11:56:21.758 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136230527}) 2024-09-18T11:56:21.759 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.760 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254871308}) 2024-09-18T11:56:21.760 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398369514}) 2024-09-18T11:56:21.760 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.760 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254871308}) 2024-09-18T11:56:21.760 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.752+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1345747586}) 2024-09-18T11:56:21.760 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.760 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462397783}) 2024-09-18T11:56:21.760 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3971641736}) 2024-09-18T11:56:21.761 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.761 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462397783}) 2024-09-18T11:56:21.761 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266526569}) 2024-09-18T11:56:21.761 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.761 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260798790}) 2024-09-18T11:56:21.762 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=815584153}) 2024-09-18T11:56:21.762 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.762 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260798790}) 2024-09-18T11:56:21.762 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384272685}) 2024-09-18T11:56:21.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T11:56:21.763 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.763 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420160796}) 2024-09-18T11:56:21.763 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3472012010}) 2024-09-18T11:56:21.763 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.764 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420160796}) 2024-09-18T11:56:21.764 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.756+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325015221}) 2024-09-18T11:56:21.764 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.764 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600414009}) 2024-09-18T11:56:21.764 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409615399}) 2024-09-18T11:56:21.764 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.764 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600414009}) 2024-09-18T11:56:21.765 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=95485001}) 2024-09-18T11:56:21.766 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.766 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771460978}) 2024-09-18T11:56:21.766 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3238884822}) 2024-09-18T11:56:21.766 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.766 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771460978}) 2024-09-18T11:56:21.766 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845323787}) 2024-09-18T11:56:21.767 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.767 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283370724}) 2024-09-18T11:56:21.767 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=117635949}) 2024-09-18T11:56:21.767 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.767 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283370724}) 2024-09-18T11:56:21.767 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.760+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612137225}) 2024-09-18T11:56:21.769 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.769 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628573400}) 2024-09-18T11:56:21.769 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1889887977}) 2024-09-18T11:56:21.769 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.769 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628573400}) 2024-09-18T11:56:21.769 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4083860141}) 2024-09-18T11:56:21.770 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.770 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523809613}) 2024-09-18T11:56:21.771 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2044160508}) 2024-09-18T11:56:21.771 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.771 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523809613}) 2024-09-18T11:56:21.771 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.764+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3735698814}) 2024-09-18T11:56:21.772 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.772 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159154993}) 2024-09-18T11:56:21.772 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=911018813}) 2024-09-18T11:56:21.772 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.772 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159154993}) 2024-09-18T11:56:21.772 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270171558}) 2024-09-18T11:56:21.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577453077}) 2024-09-18T11:56:21.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384571242}) 2024-09-18T11:56:21.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577453077}) 2024-09-18T11:56:21.773 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946101387}) 2024-09-18T11:56:21.774 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.774 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1796866593}) 2024-09-18T11:56:21.774 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970153765}) 2024-09-18T11:56:21.774 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.774 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1796866593}) 2024-09-18T11:56:21.775 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.768+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685248709}) 2024-09-18T11:56:21.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796052424}) 2024-09-18T11:56:21.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3197001113}) 2024-09-18T11:56:21.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796052424}) 2024-09-18T11:56:21.776 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955159484}) 2024-09-18T11:56:21.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465479015}) 2024-09-18T11:56:21.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612312116}) 2024-09-18T11:56:21.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465479015}) 2024-09-18T11:56:21.777 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476264949}) 2024-09-18T11:56:21.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709357037}) 2024-09-18T11:56:21.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245935514}) 2024-09-18T11:56:21.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709357037}) 2024-09-18T11:56:21.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.772+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2800287067}) 2024-09-18T11:56:21.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369986375}) 2024-09-18T11:56:21.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2771447832}) 2024-09-18T11:56:21.779 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.780 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369986375}) 2024-09-18T11:56:21.780 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3781162393}) 2024-09-18T11:56:21.781 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.781 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120453906}) 2024-09-18T11:56:21.781 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800216268}) 2024-09-18T11:56:21.781 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.782 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120453906}) 2024-09-18T11:56:21.782 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=603139740}) 2024-09-18T11:56:21.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067117011}) 2024-09-18T11:56:21.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422797953}) 2024-09-18T11:56:21.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067117011}) 2024-09-18T11:56:21.783 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.776+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660376971}) 2024-09-18T11:56:21.784 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370486820}) 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704920990}) 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370486820}) 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897376820}) 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1379133981}) 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2733632406}) 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1379133981}) 2024-09-18T11:56:21.785 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2157532171}) 2024-09-18T11:56:21.786 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.786 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246936845}) 2024-09-18T11:56:21.786 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=6816730}) 2024-09-18T11:56:21.786 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.786 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246936845}) 2024-09-18T11:56:21.786 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4107244846}) 2024-09-18T11:56:21.787 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.787 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218436598}) 2024-09-18T11:56:21.787 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3167818433}) 2024-09-18T11:56:21.787 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.787 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218436598}) 2024-09-18T11:56:21.787 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.780+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714691180}) 2024-09-18T11:56:21.788 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.788 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1518009479}) 2024-09-18T11:56:21.788 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1354927040}) 2024-09-18T11:56:21.788 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.788 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1518009479}) 2024-09-18T11:56:21.789 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2417759975}) 2024-09-18T11:56:21.789 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.789 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3070959435}) 2024-09-18T11:56:21.789 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3650082223}) 2024-09-18T11:56:21.790 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.790 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3070959435}) 2024-09-18T11:56:21.790 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371573216}) 2024-09-18T11:56:21.790 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.791 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237801111}) 2024-09-18T11:56:21.791 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2888987915}) 2024-09-18T11:56:21.791 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.791 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237801111}) 2024-09-18T11:56:21.791 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.784+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=84692315}) 2024-09-18T11:56:21.792 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.792 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016135772}) 2024-09-18T11:56:21.792 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3230206699}) 2024-09-18T11:56:21.792 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.792 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016135772}) 2024-09-18T11:56:21.792 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3542277800}) 2024-09-18T11:56:21.793 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.793 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3370730032}) 2024-09-18T11:56:21.793 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434356433}) 2024-09-18T11:56:21.793 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.793 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3370730032}) 2024-09-18T11:56:21.793 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404090784}) 2024-09-18T11:56:21.794 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.794 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728202881}) 2024-09-18T11:56:21.794 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154697908}) 2024-09-18T11:56:21.794 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.794 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728202881}) 2024-09-18T11:56:21.794 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226852446}) 2024-09-18T11:56:21.795 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.795 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045495342}) 2024-09-18T11:56:21.795 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745907376}) 2024-09-18T11:56:21.795 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.795 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.788+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045495342}) 2024-09-18T11:56:21.796 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3937434152}) 2024-09-18T11:56:21.796 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125628738}) 2024-09-18T11:56:21.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354469384}) 2024-09-18T11:56:21.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125628738}) 2024-09-18T11:56:21.797 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=812817802}) 2024-09-18T11:56:21.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848347608}) 2024-09-18T11:56:21.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2380080225}) 2024-09-18T11:56:21.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848347608}) 2024-09-18T11:56:21.798 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2835237947}) 2024-09-18T11:56:21.799 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.799 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057378782}) 2024-09-18T11:56:21.799 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4183104190}) 2024-09-18T11:56:21.799 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.799 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057378782}) 2024-09-18T11:56:21.799 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.792+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413731651}) 2024-09-18T11:56:21.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557415047}) 2024-09-18T11:56:21.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2964241797}) 2024-09-18T11:56:21.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557415047}) 2024-09-18T11:56:21.800 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912567968}) 2024-09-18T11:56:21.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322105084}) 2024-09-18T11:56:21.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2994772597}) 2024-09-18T11:56:21.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322105084}) 2024-09-18T11:56:21.801 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538025953}) 2024-09-18T11:56:21.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380839569}) 2024-09-18T11:56:21.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2393621634}) 2024-09-18T11:56:21.802 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.803 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380839569}) 2024-09-18T11:56:21.803 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4184843844}) 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381593162}) 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2635906195}) 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381593162}) 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.796+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1035901270}) 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065294428}) 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3861350266}) 2024-09-18T11:56:21.804 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065294428}) 2024-09-18T11:56:21.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033468526}) 2024-09-18T11:56:21.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059050171}) 2024-09-18T11:56:21.805 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4254357733}) 2024-09-18T11:56:21.806 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.806 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059050171}) 2024-09-18T11:56:21.806 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3654541468}) 2024-09-18T11:56:21.806 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.806 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309627168}) 2024-09-18T11:56:21.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3951738933}) 2024-09-18T11:56:21.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309627168}) 2024-09-18T11:56:21.807 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.800+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912133409}) 2024-09-18T11:56:21.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=99241907,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1834625815,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=483242162,osd_snap=80545112}) 2024-09-18T11:56:21.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=41,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_snap=24} crc {logm=3041800729,mdsmap=1531151339,mgr=2283208004,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1834625815,mon_config_key=868353578,monmap=1725997317,osd_metadata=4083148359,osd_snap=3936491675}) 2024-09-18T11:56:21.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=99241907,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1834625815,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=483242162,osd_snap=80545112}) 2024-09-18T11:56:21.808 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28} crc {logm=3627594280,mdsmap=3140514009,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=893487148,mon_config_key=858558223,monmap=1725997317,osd_metadata=3039517214,osd_pg_creating=483242162,osd_snap=257269924}) 2024-09-18T11:56:21.809 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.809 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=137796901,osdmap=1406849020}) 2024-09-18T11:56:21.809 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=909024182,osdmap=450903274}) 2024-09-18T11:56:21.809 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.809 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=137796901,osdmap=1406849020}) 2024-09-18T11:56:21.809 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=2203627652,osdmap=2814819704}) 2024-09-18T11:56:21.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3704900838}) 2024-09-18T11:56:21.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2345369356}) 2024-09-18T11:56:21.810 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.811 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3704900838}) 2024-09-18T11:56:21.811 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.804+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2896258200}) 2024-09-18T11:56:21.811 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=132394294}) 2024-09-18T11:56:21.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=768609650}) 2024-09-18T11:56:21.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=132394294}) 2024-09-18T11:56:21.812 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2724258100}) 2024-09-18T11:56:21.813 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.813 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1623762145}) 2024-09-18T11:56:21.813 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4213407735}) 2024-09-18T11:56:21.813 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.813 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1623762145}) 2024-09-18T11:56:21.813 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2578764592}) 2024-09-18T11:56:21.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1139869952}) 2024-09-18T11:56:21.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4125037210}) 2024-09-18T11:56:21.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1139869952}) 2024-09-18T11:56:21.815 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.808+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3227242557}) 2024-09-18T11:56:21.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.812+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.812+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1838144184}) 2024-09-18T11:56:21.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.812+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2219519231}) 2024-09-18T11:56:21.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.812+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.812+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1838144184}) 2024-09-18T11:56:21.817 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.812+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=913845904}) 2024-09-18T11:56:21.825 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.820+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.825 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.820+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=85} crc {osdmap=1193846956}) 2024-09-18T11:56:21.825 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.820+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=85} crc {osdmap=3790708420}) 2024-09-18T11:56:21.825 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.820+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:56:21.825 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.820+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=85} crc {osdmap=1193846956}) 2024-09-18T11:56:21.825 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:56:21.820+0000 7fbb056a9700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=90} crc {osdmap=407004212}) 2024-09-18T11:56:22.763 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:56:22.764 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:56:22.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient: get_monmap_and_config 2024-09-18T11:56:22.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient: build_initial_monmap 2024-09-18T11:56:22.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient: monmap: 2024-09-18T11:56:22.872 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:22.872 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:22.873 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:22.873102+0000 2024-09-18T11:56:22.873 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:22.873102+0000 2024-09-18T11:56:22.873 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:22.873 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:22.873 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:22.873 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:22.873 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:22.873 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:22.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 1 Processor -- start 2024-09-18T11:56:22.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 1 -- start start 2024-09-18T11:56:22.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient: init 2024-09-18T11:56:22.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:22.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient(hunting): picked mon.noname-b con 0x7f44800a21c0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient(hunting): picked mon.noname-a con 0x7f448009fdf0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient(hunting): picked mon.noname-c con 0x7f44800a01f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448006b7c0 con 0x7f448009fdf0 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44800a2590 con 0x7f44800a01f0 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448009beb0 con 0x7f44800a21c0 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f44871d3700 10 monclient(hunting): authenticate will time out at 1554.714014s 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f447ffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f448009fdf0 0x7f448013baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35930/0 (socket says 172.21.15.112:35930) 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f4484f6f700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f44800a21c0 0x7f448009f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37838/0 (socket says 172.21.15.112:37838) 2024-09-18T11:56:22.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.868+0000 7f447ffff700 1 -- 172.21.15.112:0/794804025 learned_addr learned my addr 172.21.15.112:0/794804025 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447ffff700 1 -- 172.21.15.112:0/794804025 >> v1:172.21.15.112:6789/0 conn(0x7f448009fdf0 legacy=0x7f448013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f4484f6f700 1 -- 172.21.15.112:0/794804025 >> v1:172.21.15.112:6790/0 conn(0x7f44800a21c0 legacy=0x7f448009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f4485770700 1 -- 172.21.15.112:0/794804025 >> v1:172.21.15.129:6789/0 conn(0x7f44800a01f0 legacy=0x7f448013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684739652 0 0) 0x7f448009beb0 con 0x7f44800a21c0 2024-09-18T11:56:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): my global_id is 104303 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4468003570 con 0x7f44800a21c0 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256696744 0 0) 0x7f448006b7c0 con 0x7f448009fdf0 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): my global_id is 124134 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f448009beb0 con 0x7f448009fdf0 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558574157 0 0) 0x7f44800a2590 con 0x7f44800a01f0 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient(hunting): my global_id is 114214 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f448006b7c0 con 0x7f44800a01f0 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937805729 0 0) 0x7f4468003570 con 0x7f44800a21c0 2024-09-18T11:56:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 >> v1:172.21.15.129:6789/0 conn(0x7f44800a01f0 legacy=0x7f448013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 >> v1:172.21.15.112:6789/0 conn(0x7f448009fdf0 legacy=0x7f448013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 monclient: found mon.noname-b 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44800a0fb0 con 0x7f44800a21c0 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: _finish_auth 0 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4474001a00 con 0x7f44800a21c0 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: dump: 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:22.879 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 5 monclient: authenticate success, global_id 104303 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4474001be0 con 0x7f44800a21c0 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:22.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 1 -- 172.21.15.112:0/794804025 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f44740049d0 con 0x7f44800a21c0 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f447f7fe700 10 monclient: dump: 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:22.881 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:22.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:22.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 4 set_mon_vals no callback set 2024-09-18T11:56:22.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 10 monclient: shutdown 2024-09-18T11:56:22.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 1 -- 172.21.15.112:0/794804025 >> v1:172.21.15.112:6790/0 conn(0x7f44800a21c0 legacy=0x7f448009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 1 -- 172.21.15.112:0/794804025 shutdown_connections 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 1 -- 172.21.15.112:0/794804025 >> 172.21.15.112:0/794804025 conn(0x7f44800680d0 msgr2=0x7f44800a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 1 -- 172.21.15.112:0/794804025 shutdown_connections 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.872+0000 7f44871d3700 1 -- 172.21.15.112:0/794804025 wait complete. 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient: build_initial_monmap 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient: monmap: 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 librados: starting msgr at 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 librados: starting objecter 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 Processor -- start 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 -- start start 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 librados: setting wanted keys 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 librados: calling monclient init 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient: init 2024-09-18T11:56:22.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient(hunting): picked mon.noname-b con 0x7f448009fdf0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient(hunting): picked mon.noname-c con 0x7f44800a01f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient(hunting): picked mon.noname-a con 0x7f44800a21c0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448006edd0 con 0x7f44800a21c0 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4480124c90 con 0x7f448009fdf0 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4480127090 con 0x7f44800a01f0 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f44871d3700 10 monclient(hunting): authenticate will time out at 1554.722690s 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f4485770700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f44800a21c0 0x7f448014e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35932/0 (socket says 172.21.15.112:35932) 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f447ffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f44800a01f0 0x7f4480145d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37850/0 (socket says 172.21.15.112:37850) 2024-09-18T11:56:22.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.876+0000 7f4485770700 1 -- 172.21.15.112:0/2320089150 learned_addr learned my addr 172.21.15.112:0/2320089150 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:22.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f4485770700 1 -- 172.21.15.112:0/2320089150 >> v1:172.21.15.112:6789/0 conn(0x7f44800a21c0 legacy=0x7f448014e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:22.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 client.?.objecter ms_handle_connect 0x7f44800a21c0 2024-09-18T11:56:22.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447ffff700 1 -- 172.21.15.112:0/2320089150 >> v1:172.21.15.112:6790/0 conn(0x7f44800a01f0 legacy=0x7f4480145d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 client.?.objecter ms_handle_connect 0x7f44800a01f0 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f4484f6f700 1 -- 172.21.15.112:0/2320089150 >> v1:172.21.15.129:6789/0 conn(0x7f448009fdf0 legacy=0x7f4480138c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 client.?.objecter ms_handle_connect 0x7f448009fdf0 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537161414 0 0) 0x7f448006edd0 con 0x7f44800a21c0 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): my global_id is 124137 2024-09-18T11:56:22.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f446c004f10 con 0x7f44800a21c0 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500117959 0 0) 0x7f4480127090 con 0x7f44800a01f0 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): my global_id is 104306 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f448006edd0 con 0x7f44800a01f0 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384795884 0 0) 0x7f4480124c90 con 0x7f448009fdf0 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): my global_id is 114217 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4480127090 con 0x7f448009fdf0 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1000331799 0 0) 0x7f446c004f10 con 0x7f44800a21c0 2024-09-18T11:56:22.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4480124c90 con 0x7f44800a21c0 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4478001cc0 con 0x7f44800a21c0 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): dump: 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:22.893 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2080795481 0 0) 0x7f448006edd0 con 0x7f44800a01f0 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f446c004f10 con 0x7f44800a01f0 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4470001e90 con 0x7f44800a01f0 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient(hunting): dump: 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.894 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1376260027 0 0) 0x7f4480124c90 con 0x7f44800a21c0 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 >> v1:172.21.15.112:6790/0 conn(0x7f44800a01f0 legacy=0x7f4480145d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 >> v1:172.21.15.129:6789/0 conn(0x7f448009fdf0 legacy=0x7f4480138c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient: _finish_hunting 0 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 monclient: found mon.a 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4480125e50 con 0x7f44800a21c0 2024-09-18T11:56:22.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient: _finish_auth 0 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 5 monclient: authenticate success, global_id 124137 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 10 monclient: _renew_subs 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4480126db0 con 0x7f44800a21c0 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 10 client.124137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 10 monclient: _renew_subs 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4480139530 con 0x7f44800a21c0 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f44871d3700 1 librados: init done 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44780022b0 con 0x7f44800a21c0 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:22.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4478002600 con 0x7f44800a21c0 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447effd700 4 set_mon_vals no callback set 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 10 monclient: dump: 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:22.897 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.880+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f4478002810 con 0x7f44800a21c0 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(457..457 src has 1..457) v4 ==== 5402+0+0 (unknown 1182714160 0 0) 0x7f4478043e50 con 0x7f44800a21c0 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f447cff9700 10 client.124137.objecter ms_dispatch 0x7f4480000fd0 osd_map(457..457 src has 1..457) v4 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f447cff9700 3 client.124137.objecter handle_osd_map got epochs [457,457] > 0 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f447cff9700 3 client.124137.objecter handle_osd_map decoding full epoch 457 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f447cff9700 20 client.124137.objecter dump_active .. 0 homeless 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f4484f6f700 10 monclient: get_auth_request con 0x7f446c038410 auth_method 0 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f44871d3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f44871d3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f44871d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44600050b0 con 0x7f44800a21c0 2024-09-18T11:56:22.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.884+0000 7f447cff9700 10 client.124137.objecter ms_handle_connect 0x7f446c038410 2024-09-18T11:56:22.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.896+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f4478014670 con 0x7f44800a21c0 2024-09-18T11:56:22.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.896+0000 7f447cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:22.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:22.896+0000 7f447cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:23.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:56:23.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:23.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:23.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f44600058e0 con 0x7f44800a21c0 2024-09-18T11:56:23.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f447cff9700 1 -- 172.21.15.112:0/2320089150 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1960956773) 0x7f4478014490 con 0x7f44800a21c0 2024-09-18T11:56:23.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f447cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:23.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f447cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:23.138 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 10 librados: watch_flush enter 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 10 librados: watch_flush exit 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 20 client.124137.objecter shutdown clearing up homeless session... 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 10 client.124137.objecter successfully canceled tick 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.132+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 >> v1:172.21.15.129:6812/20806 conn(0x7f446c038410 legacy=0x7f446c03a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.136+0000 7f44871d3700 10 monclient: shutdown 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.136+0000 7f44871d3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.136+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 >> v1:172.21.15.112:6789/0 conn(0x7f44800a21c0 legacy=0x7f448014e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.136+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 shutdown_connections 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.136+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 >> 172.21.15.112:0/2320089150 conn(0x7f44800680d0 msgr2=0x7f448009c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.136+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 shutdown_connections 2024-09-18T11:56:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.136+0000 7f44871d3700 1 -- 172.21.15.112:0/2320089150 wait complete. 2024-09-18T11:56:23.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.136+0000 7f44871d3700 1 librados: shutdown 2024-09-18T11:56:23.151 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:56:23.151 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T11:56:23.151 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:56:23.151 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:56:23.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient: get_monmap_and_config 2024-09-18T11:56:23.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient: monmap: 2024-09-18T11:56:23.257 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.257 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.257 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:23.257595+0000 2024-09-18T11:56:23.257 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:23.257595+0000 2024-09-18T11:56:23.257 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.258 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.258 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.258 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:23.258 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:23.258 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.258 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 1 Processor -- start 2024-09-18T11:56:23.258 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 1 -- start start 2024-09-18T11:56:23.258 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient: init 2024-09-18T11:56:23.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:56:23.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient(hunting): picked mon.noname-c con 0x7efee80a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient(hunting): picked mon.noname-a con 0x7efee80a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient(hunting): picked mon.noname-b con 0x7efee81385b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee8097270 con 0x7efee80a0750 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80a3170 con 0x7efee80a0380 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee8096cb0 con 0x7efee81385b0 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efeec984700 10 monclient(hunting): authenticate will time out at 1555.098439s 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efee759e700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7efee81385b0 0x7efee813a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37858/0 (socket says 172.21.15.112:37858) 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efee659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7efee80a0750 0x7efee80a0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35942/0 (socket says 172.21.15.112:35942) 2024-09-18T11:56:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.252+0000 7efee759e700 1 -- 172.21.15.112:0/862461504 learned_addr learned my addr 172.21.15.112:0/862461504 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee659c700 1 -- 172.21.15.112:0/862461504 >> v1:172.21.15.112:6789/0 conn(0x7efee80a0750 legacy=0x7efee80a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee6d9d700 1 -- 172.21.15.112:0/862461504 >> v1:172.21.15.129:6789/0 conn(0x7efee80a0380 legacy=0x7efee8097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee759e700 1 -- 172.21.15.112:0/862461504 >> v1:172.21.15.112:6790/0 conn(0x7efee81385b0 legacy=0x7efee813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554591059 0 0) 0x7efee8097270 con 0x7efee80a0750 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): my global_id is 124140 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0003570 con 0x7efee80a0750 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886132642 0 0) 0x7efee80a3170 con 0x7efee80a0380 2024-09-18T11:56:23.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): my global_id is 114220 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee8097270 con 0x7efee80a0380 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192349519 0 0) 0x7efee8096cb0 con 0x7efee81385b0 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient(hunting): my global_id is 104309 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee80a3170 con 0x7efee81385b0 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944909602 0 0) 0x7efed0003570 con 0x7efee80a0750 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 >> v1:172.21.15.112:6790/0 conn(0x7efee81385b0 legacy=0x7efee813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 >> v1:172.21.15.129:6789/0 conn(0x7efee80a0380 legacy=0x7efee8097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 monclient: found mon.noname-a 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80a1420 con 0x7efee80a0750 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 5 monclient: authenticate success, global_id 124140 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efedc001a00 con 0x7efee80a0750 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: dump: 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.265 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efedc001be0 con 0x7efee80a0750 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 4 set_mon_vals no callback set 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 1 -- 172.21.15.112:0/862461504 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efedc0049d0 con 0x7efee80a0750 2024-09-18T11:56:23.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efee5d9b700 10 monclient: dump: 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 10 monclient: shutdown 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 1 -- 172.21.15.112:0/862461504 >> v1:172.21.15.112:6789/0 conn(0x7efee80a0750 legacy=0x7efee80a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 1 -- 172.21.15.112:0/862461504 shutdown_connections 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 1 -- 172.21.15.112:0/862461504 >> 172.21.15.112:0/862461504 conn(0x7efee80a5990 msgr2=0x7efee80a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 1 -- 172.21.15.112:0/862461504 shutdown_connections 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 1 -- 172.21.15.112:0/862461504 wait complete. 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 10 monclient: monmap: 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.268 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:23.269 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:23.269 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 1 librados: starting msgr at 2024-09-18T11:56:23.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 1 librados: starting objecter 2024-09-18T11:56:23.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.256+0000 7efeec984700 1 Processor -- start 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 1 -- start start 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 1 librados: setting wanted keys 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 1 librados: calling monclient init 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 10 monclient: init 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 10 monclient(hunting): picked mon.noname-b con 0x7efee80a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 10 monclient(hunting): picked mon.noname-c con 0x7efee80a0750 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 10 monclient(hunting): picked mon.noname-a con 0x7efee81385b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee81424a0 con 0x7efee81385b0 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee81389f0 con 0x7efee80a0380 2024-09-18T11:56:23.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee8098740 con 0x7efee80a0750 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeec984700 10 monclient(hunting): authenticate will time out at 1555.104797s 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efee659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7efee80a0750 0x7efee809df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37866/0 (socket says 172.21.15.112:37866) 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efee659c700 1 -- 172.21.15.112:0/165573161 learned_addr learned my addr 172.21.15.112:0/165573161 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efee659c700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6790/0 conn(0x7efee80a0750 legacy=0x7efee809df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 client.?.objecter ms_handle_connect 0x7efee80a0750 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efee759e700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6789/0 conn(0x7efee81385b0 legacy=0x7efee8149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 client.?.objecter ms_handle_connect 0x7efee81385b0 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633040896 0 0) 0x7efee8098740 con 0x7efee80a0750 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): my global_id is 104312 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed4004f10 con 0x7efee80a0750 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089217056 0 0) 0x7efee81424a0 con 0x7efee81385b0 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): my global_id is 124143 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee8098740 con 0x7efee81385b0 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 client.?.objecter ms_handle_connect 0x7efee80a0380 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efee6d9d700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.129:6789/0 conn(0x7efee80a0380 legacy=0x7efee8141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1462418208 0 0) 0x7efed4004f10 con 0x7efee80a0750 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee81424a0 con 0x7efee80a0750 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767011413 0 0) 0x7efee81389f0 con 0x7efee80a0380 2024-09-18T11:56:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): my global_id is 114223 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed4004f10 con 0x7efee80a0380 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efedc0017a0 con 0x7efee80a0750 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): dump: 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.278 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.260+0000 7efeceffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1149838293 0 0) 0x7efee8098740 con 0x7efee81385b0 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee81389f0 con 0x7efee81385b0 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efed8001bf0 con 0x7efee81385b0 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient(hunting): dump: 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.279 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 400947453 0 0) 0x7efee81424a0 con 0x7efee80a0750 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.129:6789/0 conn(0x7efee80a0380 legacy=0x7efee8141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6789/0 conn(0x7efee81385b0 legacy=0x7efee8149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 monclient: found mon.c 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8098460 con 0x7efee80a0750 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 5 monclient: authenticate success, global_id 104312 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 10 monclient: _renew_subs 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee8141c80 con 0x7efee80a0750 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 10 client.104312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 10 monclient: _renew_subs 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efee8137ea0 con 0x7efee80a0750 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 1 librados: init done 2024-09-18T11:56:23.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efedc005030 con 0x7efee80a0750 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efedc005340 con 0x7efee80a0750 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efee4d99700 4 set_mon_vals no callback set 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 monclient: dump: 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.282 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efec8006590 con 0x7efec8002840 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7efedc00fbc0 con 0x7efee80a0750 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(457..457 src has 1..457) v4 ==== 5402+0+0 (unknown 1182714160 0 0) 0x7efedc043fc0 con 0x7efee80a0750 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 10 client.104312.objecter ms_dispatch 0x7efee8000fd0 osd_map(457..457 src has 1..457) v4 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 3 client.104312.objecter handle_osd_map got epochs [457,457] > 0 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 3 client.104312.objecter handle_osd_map decoding full epoch 457 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efee759e700 10 monclient: get_auth_request con 0x7efed4038580 auth_method 0 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.264+0000 7efeceffd700 20 client.104312.objecter dump_active .. 0 homeless 2024-09-18T11:56:23.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efee6d9d700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6789/0 conn(0x7efec8002840 legacy=0x7efec8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 client.104312.objecter ms_handle_connect 0x7efec8002840 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 client.104312.objecter resend_mon_ops 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643997281 0 0) 0x7efec8006590 con 0x7efec8002840 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 monclient(hunting): my global_id is 124146 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee81424a0 con 0x7efec8002840 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 748723358 0 0) 0x7efee81424a0 con 0x7efec8002840 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efec8006590 con 0x7efec8002840 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efee0001e80 con 0x7efec8002840 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 monclient: dump: 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 386667664 0 0) 0x7efec8006590 con 0x7efec8002840 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efec800a140 con 0x7efec8002840 2024-09-18T11:56:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7efee0005b90 con 0x7efec8002840 2024-09-18T11:56:23.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:23.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6789/0 conn(0x7efec8002840 legacy=0x7efec8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6789/0 conn(0x7efec8002840 legacy=0x7efec8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:23.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.268+0000 7efeceffd700 10 client.104312.objecter ms_handle_connect 0x7efed4038580 2024-09-18T11:56:23.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.276+0000 7efeec984700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:56:23.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.276+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efec8009670 con 0x7efec80090e0 2024-09-18T11:56:23.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 client.104312.objecter ms_handle_connect 0x7efec80090e0 2024-09-18T11:56:23.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 client.104312.objecter resend_mon_ops 2024-09-18T11:56:23.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efee6d9d700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6789/0 conn(0x7efec80090e0 legacy=0x7efec8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055075875 0 0) 0x7efec8009670 con 0x7efec80090e0 2024-09-18T11:56:23.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 monclient(hunting): my global_id is 124149 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec8006590 con 0x7efec80090e0 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2254342916 0 0) 0x7efec8006590 con 0x7efec80090e0 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efec8009670 con 0x7efec80090e0 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7efee0002650 con 0x7efec80090e0 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 monclient: dump: 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4021042276 0 0) 0x7efec8009670 con 0x7efec80090e0 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efec8009930 con 0x7efec80090e0 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 1212405828) 0x7efee0002920 con 0x7efec80090e0 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6789/0 conn(0x7efec80090e0 legacy=0x7efec8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeceffd700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6789/0 conn(0x7efec80090e0 legacy=0x7efec8008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 54, 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:56:23.290 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.291 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:56:23.293 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.294 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.295 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.296 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:56:23.297 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.298 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:56:23.299 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.301 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:56:23.301 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 10 librados: watch_flush enter 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 10 librados: watch_flush exit 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 20 client.104312.objecter shutdown clearing up homeless session... 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 10 client.104312.objecter successfully canceled tick 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.129:6812/20806 conn(0x7efed4038580 legacy=0x7efed403aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 10 monclient: shutdown 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 >> v1:172.21.15.112:6790/0 conn(0x7efee80a0750 legacy=0x7efee809df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efee759e700 1 -- 172.21.15.112:0/165573161 reap_dead start 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 shutdown_connections 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 >> 172.21.15.112:0/165573161 conn(0x7efee80a5990 msgr2=0x7efee8138c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 shutdown_connections 2024-09-18T11:56:23.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 1 -- 172.21.15.112:0/165573161 wait complete. 2024-09-18T11:56:23.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.280+0000 7efeec984700 1 librados: shutdown 2024-09-18T11:56:23.305 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:56:23.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:56:23.305 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:56:23.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.408+0000 7f482359e700 10 monclient: get_monmap_and_config 2024-09-18T11:56:23.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.408+0000 7f482359e700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.408+0000 7f482359e700 10 monclient: monmap: 2024-09-18T11:56:23.415 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.415 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.416 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:23.416067+0000 2024-09-18T11:56:23.416 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:23.416067+0000 2024-09-18T11:56:23.416 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.416 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.416 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.416 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:23.416 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:23.416 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 1 Processor -- start 2024-09-18T11:56:23.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 1 -- start start 2024-09-18T11:56:23.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 10 monclient: init 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 10 monclient(hunting): picked mon.noname-c con 0x7f48240a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 10 monclient(hunting): picked mon.noname-b con 0x7f4824096a60 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 10 monclient(hunting): picked mon.noname-a con 0x7f482413ca20 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482406e910 con 0x7f482413ca20 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482406e630 con 0x7f48240a1ea0 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240a2270 con 0x7f4824096a60 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482359e700 10 monclient(hunting): authenticate will time out at 1555.256892s 2024-09-18T11:56:23.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f4821d9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f4824096a60 0x7f482406c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37874/0 (socket says 172.21.15.112:37874) 2024-09-18T11:56:23.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f4821d9b700 1 -- 172.21.15.112:0/1700853752 learned_addr learned my addr 172.21.15.112:0/1700853752 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f4821d9b700 1 -- 172.21.15.112:0/1700853752 >> v1:172.21.15.112:6790/0 conn(0x7f4824096a60 legacy=0x7f482406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f4822d9d700 1 -- 172.21.15.112:0/1700853752 >> v1:172.21.15.112:6789/0 conn(0x7f482413ca20 legacy=0x7f482413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482259c700 1 -- 172.21.15.112:0/1700853752 >> v1:172.21.15.129:6789/0 conn(0x7f48240a1ea0 legacy=0x7f482406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442707320 0 0) 0x7f482406e910 con 0x7f482413ca20 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 10 monclient(hunting): my global_id is 124152 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480c003570 con 0x7f482413ca20 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584547404 0 0) 0x7f48240a2270 con 0x7f4824096a60 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 10 monclient(hunting): my global_id is 104315 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.412+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482406e910 con 0x7f4824096a60 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463689169 0 0) 0x7f482406e630 con 0x7f48240a1ea0 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient(hunting): my global_id is 114226 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48240a2270 con 0x7f48240a1ea0 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591128136 0 0) 0x7f480c003570 con 0x7f482413ca20 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 >> v1:172.21.15.112:6790/0 conn(0x7f4824096a60 legacy=0x7f482406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 >> v1:172.21.15.129:6789/0 conn(0x7f48240a1ea0 legacy=0x7f482406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 monclient: found mon.noname-a 2024-09-18T11:56:23.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482406eed0 con 0x7f482413ca20 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f481c001990 con 0x7f482413ca20 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 5 monclient: authenticate success, global_id 124152 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: dump: 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.423 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f481c001c20 con 0x7f482413ca20 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 1 -- 172.21.15.112:0/1700853752 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f481c0020a0 con 0x7f482413ca20 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482159a700 10 monclient: dump: 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.425 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 4 set_mon_vals no callback set 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient: shutdown 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- 172.21.15.112:0/1700853752 >> v1:172.21.15.112:6789/0 conn(0x7f482413ca20 legacy=0x7f482413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- 172.21.15.112:0/1700853752 shutdown_connections 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- 172.21.15.112:0/1700853752 >> 172.21.15.112:0/1700853752 conn(0x7f4824067da0 msgr2=0x7f48240a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- 172.21.15.112:0/1700853752 shutdown_connections 2024-09-18T11:56:23.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- 172.21.15.112:0/1700853752 wait complete. 2024-09-18T11:56:23.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient: monmap: 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 librados: starting msgr at 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 librados: starting objecter 2024-09-18T11:56:23.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 Processor -- start 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- start start 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 librados: setting wanted keys 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 librados: calling monclient init 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient: init 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient(hunting): picked mon.noname-b con 0x7f4824096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient(hunting): picked mon.noname-c con 0x7f48240a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient(hunting): picked mon.noname-a con 0x7f482413ca20 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4824125350 con 0x7f482413ca20 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482409dde0 con 0x7f4824096a60 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4824127b20 con 0x7f48240a1ea0 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f482359e700 10 monclient(hunting): authenticate will time out at 1555.263623s 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f4822d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f482413ca20 0x7f482414e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36012/0 (socket says 172.21.15.112:36012) 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f4821d9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f48240a1ea0 0x7f4824146590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37886/0 (socket says 172.21.15.112:37886) 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.416+0000 7f4822d9d700 1 -- 172.21.15.112:0/3716301883 learned_addr learned my addr 172.21.15.112:0/3716301883 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f4822d9d700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6789/0 conn(0x7f482413ca20 legacy=0x7f482414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 client.?.objecter ms_handle_connect 0x7f482413ca20 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 client.?.objecter ms_handle_connect 0x7f48240a1ea0 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f4821d9b700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6790/0 conn(0x7f48240a1ea0 legacy=0x7f4824146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482259c700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.129:6789/0 conn(0x7f4824096a60 legacy=0x7f4824145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 client.?.objecter ms_handle_connect 0x7f4824096a60 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056443175 0 0) 0x7f4824125350 con 0x7f482413ca20 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): my global_id is 124155 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4804004f10 con 0x7f482413ca20 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100675577 0 0) 0x7f482409dde0 con 0x7f4824096a60 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): my global_id is 114229 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4824125350 con 0x7f4824096a60 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3675701774 0 0) 0x7f4804004f10 con 0x7f482413ca20 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f482409dde0 con 0x7f482413ca20 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949303746 0 0) 0x7f4824127b20 con 0x7f48240a1ea0 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): my global_id is 104318 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4804004f10 con 0x7f48240a1ea0 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f481c0020a0 con 0x7f482413ca20 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): dump: 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.437 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4115557322 0 0) 0x7f482409dde0 con 0x7f482413ca20 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6790/0 conn(0x7f48240a1ea0 legacy=0x7f4824146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.129:6789/0 conn(0x7f4824096a60 legacy=0x7f4824145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 monclient: found mon.a 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824146ca0 con 0x7f482413ca20 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 5 monclient: authenticate success, global_id 124155 2024-09-18T11:56:23.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 10 monclient: _renew_subs 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482406e110 con 0x7f482413ca20 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 10 client.124155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 10 monclient: _renew_subs 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48241259a0 con 0x7f482413ca20 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f482359e700 1 librados: init done 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f481c002650 con 0x7f482413ca20 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f481c002840 con 0x7f482413ca20 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f4820d99700 4 set_mon_vals no callback set 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.420+0000 7f48127fc700 10 monclient: dump: 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f481c0029a0 con 0x7f482413ca20 2024-09-18T11:56:23.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f482359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4808006590 con 0x7f4808002840 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(457..457 src has 1..457) v4 ==== 5402+0+0 (unknown 1182714160 0 0) 0x7f481c043f50 con 0x7f482413ca20 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 client.124155.objecter ms_dispatch 0x7f4824000fd0 osd_map(457..457 src has 1..457) v4 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 3 client.124155.objecter handle_osd_map got epochs [457,457] > 0 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 3 client.124155.objecter handle_osd_map decoding full epoch 457 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 20 client.124155.objecter dump_active .. 0 homeless 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f4821d9b700 10 monclient: get_auth_request con 0x7f48040383d0 auth_method 0 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 client.124155.objecter ms_handle_connect 0x7f4808002840 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 client.124155.objecter resend_mon_ops 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f482259c700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6790/0 conn(0x7f4808002840 legacy=0x7f4808004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098734880 0 0) 0x7f4808006590 con 0x7f4808002840 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 monclient(hunting): my global_id is 104321 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482409dde0 con 0x7f4808002840 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 499460868 0 0) 0x7f482409dde0 con 0x7f4808002840 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4808006590 con 0x7f4808002840 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 client.124155.objecter ms_handle_connect 0x7f48040383d0 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4814001e50 con 0x7f4808002840 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 10 monclient: dump: 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.442 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 987234481 0 0) 0x7f4808006590 con 0x7f4808002840 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.424+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f480800a140 con 0x7f4808002840 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.428+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f4814005b90 con 0x7f4808002840 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.428+0000 7f48127fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.428+0000 7f48127fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:23.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.428+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6790/0 conn(0x7f4808002840 legacy=0x7f4808004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.428+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6790/0 conn(0x7f4808002840 legacy=0x7f4808004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:23.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.436+0000 7f482359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.436+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4808007520 con 0x7f48080068d0 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 client.124155.objecter ms_handle_connect 0x7f48080068d0 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 client.124155.objecter resend_mon_ops 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482259c700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6790/0 conn(0x7f48080068d0 legacy=0x7f4808006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475638633 0 0) 0x7f4808007520 con 0x7f48080068d0 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 monclient(hunting): my global_id is 104324 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4808006590 con 0x7f48080068d0 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3588292928 0 0) 0x7f4808006590 con 0x7f48080068d0 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4808007520 con 0x7f48080068d0 2024-09-18T11:56:23.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4814002610 con 0x7f48080068d0 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 monclient: dump: 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 552691516 0 0) 0x7f4808007520 con 0x7f48080068d0 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4808008760 con 0x7f48080068d0 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 2735200145) 0x7f48140028c0 con 0x7f48080068d0 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6790/0 conn(0x7f48080068d0 legacy=0x7f4808006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f48127fc700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6790/0 conn(0x7f48080068d0 legacy=0x7f4808006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:23.448 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 54, 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.449 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.450 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:56:23.451 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.452 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:23.453 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.454 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.455 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:23.456 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.457 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:56:23.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.460 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:56:23.460 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:23.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 10 librados: watch_flush enter 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 10 librados: watch_flush exit 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 20 client.124155.objecter shutdown clearing up homeless session... 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 10 client.124155.objecter successfully canceled tick 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.129:6812/20806 conn(0x7f48040383d0 legacy=0x7f480403a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 10 monclient: shutdown 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f4822d9d700 1 -- 172.21.15.112:0/3716301883 reap_dead start 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 >> v1:172.21.15.112:6789/0 conn(0x7f482413ca20 legacy=0x7f482414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 shutdown_connections 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 >> 172.21.15.112:0/3716301883 conn(0x7f4824067da0 msgr2=0x7f482409fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 shutdown_connections 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 1 -- 172.21.15.112:0/3716301883 wait complete. 2024-09-18T11:56:23.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.440+0000 7f482359e700 1 librados: shutdown 2024-09-18T11:56:23.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:56:23.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:56:23.463 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 10 monclient: get_monmap_and_config 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 10 monclient: monmap: 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:23.574616+0000 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:23.574616+0000 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:23.574 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:23.575 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 1 Processor -- start 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 1 -- start start 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 10 monclient: init 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 10 monclient(hunting): picked mon.noname-a con 0x7fe3700a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.568+0000 7fe376e13700 10 monclient(hunting): picked mon.noname-c con 0x7fe37012edc0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 10 monclient(hunting): picked mon.noname-b con 0x7fe37012f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3700981a0 con 0x7fe3700a2040 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37006c120 con 0x7fe37012edc0 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37006eef0 con 0x7fe37012f8a0 2024-09-18T11:56:23.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 10 monclient(hunting): authenticate will time out at 1555.416102s 2024-09-18T11:56:23.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376612700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe37012f8a0 0x7fe370137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37916/0 (socket says 172.21.15.112:37916) 2024-09-18T11:56:23.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe375e11700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe3700a2040 0x7fe370097a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36024/0 (socket says 172.21.15.112:36024) 2024-09-18T11:56:23.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376612700 1 -- 172.21.15.112:0/4063207782 learned_addr learned my addr 172.21.15.112:0/4063207782 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe375e11700 1 -- 172.21.15.112:0/4063207782 >> v1:172.21.15.112:6789/0 conn(0x7fe3700a2040 legacy=0x7fe370097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376612700 1 -- 172.21.15.112:0/4063207782 >> v1:172.21.15.112:6790/0 conn(0x7fe37012f8a0 legacy=0x7fe370137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe375610700 1 -- 172.21.15.112:0/4063207782 >> v1:172.21.15.129:6789/0 conn(0x7fe37012edc0 legacy=0x7fe37012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190706258 0 0) 0x7fe37006eef0 con 0x7fe37012f8a0 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): my global_id is 104327 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe35c003570 con 0x7fe37012f8a0 2024-09-18T11:56:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420392955 0 0) 0x7fe3700981a0 con 0x7fe3700a2040 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): my global_id is 124158 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe37006eef0 con 0x7fe3700a2040 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361707344 0 0) 0x7fe37006c120 con 0x7fe37012edc0 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient(hunting): my global_id is 114232 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3700981a0 con 0x7fe37012edc0 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 772053051 0 0) 0x7fe35c003570 con 0x7fe37012f8a0 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 >> v1:172.21.15.129:6789/0 conn(0x7fe37012edc0 legacy=0x7fe37012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 >> v1:172.21.15.112:6789/0 conn(0x7fe3700a2040 legacy=0x7fe370097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 monclient: found mon.noname-b 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe370096dd0 con 0x7fe37012f8a0 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe368001cc0 con 0x7fe37012f8a0 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: dump: 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.592 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 5 monclient: authenticate success, global_id 104327 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe368001f50 con 0x7fe37012f8a0 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 1 -- 172.21.15.112:0/4063207782 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe368004e10 con 0x7fe37012f8a0 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe376e13700 4 set_mon_vals no callback set 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.572+0000 7fe374e0f700 10 monclient: dump: 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.594 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient: shutdown 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- 172.21.15.112:0/4063207782 >> v1:172.21.15.112:6790/0 conn(0x7fe37012f8a0 legacy=0x7fe370137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- 172.21.15.112:0/4063207782 shutdown_connections 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- 172.21.15.112:0/4063207782 >> 172.21.15.112:0/4063207782 conn(0x7fe370067da0 msgr2=0x7fe3700a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- 172.21.15.112:0/4063207782 shutdown_connections 2024-09-18T11:56:23.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- 172.21.15.112:0/4063207782 wait complete. 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient: monmap: 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 librados: starting msgr at 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 librados: starting objecter 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 Processor -- start 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- start start 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 librados: setting wanted keys 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 librados: calling monclient init 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient: init 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient(hunting): picked mon.noname-c con 0x7fe3700a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient(hunting): picked mon.noname-b con 0x7fe37012edc0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient(hunting): picked mon.noname-a con 0x7fe37012f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37009da00 con 0x7fe37012f8a0 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37009e130 con 0x7fe37012edc0 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37006f110 con 0x7fe3700a2040 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376e13700 10 monclient(hunting): authenticate will time out at 1555.423118s 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe375e11700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe3700a2040 0x7fe37009b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37920/0 (socket says 172.21.15.112:37920) 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe376612700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe37012f8a0 0x7fe37009c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36028/0 (socket says 172.21.15.112:36028) 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.576+0000 7fe375e11700 1 -- 172.21.15.112:0/402283619 learned_addr learned my addr 172.21.15.112:0/402283619 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376612700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.112:6789/0 conn(0x7fe37012f8a0 legacy=0x7fe37009c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 client.?.objecter ms_handle_connect 0x7fe37012f8a0 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 client.?.objecter ms_handle_connect 0x7fe3700a2040 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe375e11700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.112:6790/0 conn(0x7fe3700a2040 legacy=0x7fe37009b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 client.?.objecter ms_handle_connect 0x7fe37012edc0 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe375610700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6789/0 conn(0x7fe37012edc0 legacy=0x7fe37009bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691857474 0 0) 0x7fe37006f110 con 0x7fe3700a2040 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): my global_id is 104330 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe354004f10 con 0x7fe3700a2040 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922578779 0 0) 0x7fe37009da00 con 0x7fe37012f8a0 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): my global_id is 124161 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe37006f110 con 0x7fe37012f8a0 2024-09-18T11:56:23.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963063104 0 0) 0x7fe37009e130 con 0x7fe37012edc0 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): my global_id is 114235 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe37009da00 con 0x7fe37012edc0 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4043535315 0 0) 0x7fe354004f10 con 0x7fe3700a2040 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe37009e130 con 0x7fe3700a2040 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe36c001cd0 con 0x7fe3700a2040 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): dump: 2024-09-18T11:56:23.600 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2488665234 0 0) 0x7fe37009da00 con 0x7fe37012edc0 2024-09-18T11:56:23.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe354004f10 con 0x7fe37012edc0 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe360001b80 con 0x7fe37012edc0 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): dump: 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.602 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1324085821 0 0) 0x7fe37006f110 con 0x7fe37012f8a0 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe37009da00 con 0x7fe37012f8a0 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe3680016e0 con 0x7fe37012f8a0 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient(hunting): dump: 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.603 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3546775327 0 0) 0x7fe37009e130 con 0x7fe3700a2040 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6789/0 conn(0x7fe37012edc0 legacy=0x7fe37009bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.112:6789/0 conn(0x7fe37012f8a0 legacy=0x7fe37009c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 monclient: found mon.c 2024-09-18T11:56:23.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3701187b0 con 0x7fe3700a2040 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 5 monclient: authenticate success, global_id 104330 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 10 monclient: _renew_subs 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37009d0c0 con 0x7fe3700a2040 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 10 client.104330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 10 monclient: _renew_subs 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37009dbd0 con 0x7fe3700a2040 2024-09-18T11:56:23.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe376e13700 1 librados: init done 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe36c004c50 con 0x7fe3700a2040 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe36c004fe0 con 0x7fe3700a2040 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe367fff700 4 set_mon_vals no callback set 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.580+0000 7fe365ffb700 10 monclient: dump: 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.606 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe376e13700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe350006590 con 0x7fe350002840 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe36c005140 con 0x7fe3700a2040 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(457..457 src has 1..457) v4 ==== 5402+0+0 (unknown 1182714160 0 0) 0x7fe36c043f60 con 0x7fe3700a2040 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 10 client.104330.objecter ms_dispatch 0x7fe370000fd0 osd_map(457..457 src has 1..457) v4 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 3 client.104330.objecter handle_osd_map got epochs [457,457] > 0 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 3 client.104330.objecter handle_osd_map decoding full epoch 457 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 20 client.104330.objecter dump_active .. 0 homeless 2024-09-18T11:56:23.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe376612700 10 monclient: get_auth_request con 0x7fe354038660 auth_method 0 2024-09-18T11:56:23.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 10 client.104330.objecter ms_handle_connect 0x7fe350002840 2024-09-18T11:56:23.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 10 client.104330.objecter resend_mon_ops 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe375610700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6789/0 conn(0x7fe350002840 legacy=0x7fe350004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703933266 0 0) 0x7fe350006590 con 0x7fe350002840 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 10 monclient(hunting): my global_id is 114238 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.584+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe37009e130 con 0x7fe350002840 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3638366983 0 0) 0x7fe37009e130 con 0x7fe350002840 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe350006590 con 0x7fe350002840 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe360002310 con 0x7fe350002840 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:23.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 10 monclient: dump: 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 10 client.104330.objecter ms_handle_connect 0x7fe354038660 2024-09-18T11:56:23.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 797075199 0 0) 0x7fe350006590 con 0x7fe350002840 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe35000a140 con 0x7fe350002840 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe360005810 con 0x7fe350002840 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6789/0 conn(0x7fe350002840 legacy=0x7fe350004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.588+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6789/0 conn(0x7fe350002840 legacy=0x7fe350004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.600+0000 7fe376e13700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.600+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe350007550 con 0x7fe350006900 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.600+0000 7fe375610700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6789/0 conn(0x7fe350006900 legacy=0x7fe350006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.600+0000 7fe365ffb700 10 client.104330.objecter ms_handle_connect 0x7fe350006900 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.600+0000 7fe365ffb700 10 client.104330.objecter resend_mon_ops 2024-09-18T11:56:23.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050872513 0 0) 0x7fe350007550 con 0x7fe350006900 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 10 monclient(hunting): my global_id is 114241 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe350006590 con 0x7fe350006900 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 15370633 0 0) 0x7fe350006590 con 0x7fe350006900 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe350007550 con 0x7fe350006900 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe360002880 con 0x7fe350006900 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 10 monclient: dump: 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.614 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3060455798 0 0) 0x7fe350007550 con 0x7fe350006900 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe3500087b0 con 0x7fe350006900 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 2754667504) 0x7fe360002aa0 con 0x7fe350006900 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:23.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6789/0 conn(0x7fe350006900 legacy=0x7fe350006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe365ffb700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6789/0 conn(0x7fe350006900 legacy=0x7fe350006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:23.616 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:23.616 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:23.616 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:23.616 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:56:23.616 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 54, 2024-09-18T11:56:23.616 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.617 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.618 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:56:23.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.622 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.623 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:56:23.625 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.626 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.627 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:56:23.628 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:23.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 10 librados: watch_flush enter 2024-09-18T11:56:23.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 10 librados: watch_flush exit 2024-09-18T11:56:23.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 20 client.104330.objecter shutdown clearing up homeless session... 2024-09-18T11:56:23.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 10 client.104330.objecter successfully canceled tick 2024-09-18T11:56:23.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.129:6812/20806 conn(0x7fe354038660 legacy=0x7fe35403ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 10 monclient: shutdown 2024-09-18T11:56:23.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 >> v1:172.21.15.112:6790/0 conn(0x7fe3700a2040 legacy=0x7fe37009b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376612700 1 -- 172.21.15.112:0/402283619 reap_dead start 2024-09-18T11:56:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 shutdown_connections 2024-09-18T11:56:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 >> 172.21.15.112:0/402283619 conn(0x7fe370067da0 msgr2=0x7fe37006c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 shutdown_connections 2024-09-18T11:56:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 1 -- 172.21.15.112:0/402283619 wait complete. 2024-09-18T11:56:23.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.604+0000 7fe376e13700 1 librados: shutdown 2024-09-18T11:56:23.631 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:56:23.631 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T11:56:23.632 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T11:56:23.632 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T11:56:23.632 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-18T11:56:23.632 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.736+0000 7f1ebbfff700 10 monclient: get_monmap_and_config 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.736+0000 7f1ebbfff700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.736+0000 7f1ebbfff700 10 monclient: monmap: 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:23.743836+0000 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:23.743836+0000 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.745 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:23.746 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 1 Processor -- start 2024-09-18T11:56:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 1 -- start start 2024-09-18T11:56:23.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 10 monclient: init 2024-09-18T11:56:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f1ebc0961d0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f1ebc098630 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f1ebc135530 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc06a200 con 0x7f1ebc098630 2024-09-18T11:56:23.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc06fcd0 con 0x7f1ebc135530 2024-09-18T11:56:23.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc0973c0 con 0x7f1ebc0961d0 2024-09-18T11:56:23.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebaffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1ebc0961d0 0x7f1ebc0965a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37928/0 (socket says 172.21.15.112:37928) 2024-09-18T11:56:23.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebaffd700 1 -- 172.21.15.112:0/264156817 learned_addr learned my addr 172.21.15.112:0/264156817 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.740+0000 7f1ebbfff700 10 monclient(hunting): authenticate will time out at 1555.586319s 2024-09-18T11:56:23.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebaffd700 1 -- 172.21.15.112:0/264156817 >> v1:172.21.15.112:6790/0 conn(0x7f1ebc0961d0 legacy=0x7f1ebc0965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eba7fc700 1 -- 172.21.15.112:0/264156817 >> v1:172.21.15.112:6789/0 conn(0x7f1ebc098630 legacy=0x7f1ebc096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990197357 0 0) 0x7f1ebc0973c0 con 0x7f1ebc0961d0 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): my global_id is 104333 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea4003570 con 0x7f1ebc0961d0 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebb7fe700 1 -- 172.21.15.112:0/264156817 >> v1:172.21.15.129:6789/0 conn(0x7f1ebc135530 legacy=0x7f1ebc137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775587454 0 0) 0x7f1ebc06a200 con 0x7f1ebc098630 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): my global_id is 124164 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc0973c0 con 0x7f1ebc098630 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718833225 0 0) 0x7f1ebc06fcd0 con 0x7f1ebc135530 2024-09-18T11:56:23.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient(hunting): my global_id is 114244 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc06a200 con 0x7f1ebc135530 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 427483354 0 0) 0x7f1ea4003570 con 0x7f1ebc0961d0 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 >> v1:172.21.15.129:6789/0 conn(0x7f1ebc135530 legacy=0x7f1ebc137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 >> v1:172.21.15.112:6789/0 conn(0x7f1ebc098630 legacy=0x7f1ebc096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 monclient: found mon.noname-b 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc063b20 con 0x7f1ebc0961d0 2024-09-18T11:56:23.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 5 monclient: authenticate success, global_id 104333 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1eac001930 con 0x7f1ebc0961d0 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: dump: 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.754 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eac001bc0 con 0x7f1ebc0961d0 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 1 -- 172.21.15.112:0/264156817 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1eac004a90 con 0x7f1ebc0961d0 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 4 set_mon_vals no callback set 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1eb9ffb700 10 monclient: dump: 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 10 monclient: shutdown 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 1 -- 172.21.15.112:0/264156817 >> v1:172.21.15.112:6790/0 conn(0x7f1ebc0961d0 legacy=0x7f1ebc0965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 1 -- 172.21.15.112:0/264156817 shutdown_connections 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 1 -- 172.21.15.112:0/264156817 >> 172.21.15.112:0/264156817 conn(0x7f1ebc0a2190 msgr2=0x7f1ebc0a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 1 -- 172.21.15.112:0/264156817 shutdown_connections 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 1 -- 172.21.15.112:0/264156817 wait complete. 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 10 monclient: monmap: 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.757 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 1 librados: starting msgr at 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.744+0000 7f1ebbfff700 1 librados: starting objecter 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 1 Processor -- start 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 1 -- start start 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 1 librados: setting wanted keys 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 1 librados: calling monclient init 2024-09-18T11:56:23.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 10 monclient: init 2024-09-18T11:56:23.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:23.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f1ebc0961d0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f1ebc098630 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f1ebc135530 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc06fac0 con 0x7f1ebc098630 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc09c050 con 0x7f1ebc135530 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc08dd90 con 0x7f1ebc0961d0 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebbfff700 10 monclient(hunting): authenticate will time out at 1555.593330s 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebaffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1ebc0961d0 0x7f1ebc08cf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37932/0 (socket says 172.21.15.112:37932) 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebaffd700 1 -- 172.21.15.112:0/2664651238 learned_addr learned my addr 172.21.15.112:0/2664651238 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebaffd700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6790/0 conn(0x7f1ebc0961d0 legacy=0x7f1ebc08cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 client.?.objecter ms_handle_connect 0x7f1ebc0961d0 2024-09-18T11:56:23.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1eba7fc700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6789/0 conn(0x7f1ebc098630 legacy=0x7f1ebc08d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 client.?.objecter ms_handle_connect 0x7f1ebc098630 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352774019 0 0) 0x7f1ebc08dd90 con 0x7f1ebc0961d0 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 monclient(hunting): my global_id is 104336 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8004f10 con 0x7f1ebc0961d0 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 client.?.objecter ms_handle_connect 0x7f1ebc135530 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ebb7fe700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.129:6789/0 conn(0x7f1ebc135530 legacy=0x7f1ebc147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.748+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644871334 0 0) 0x7f1ebc06fac0 con 0x7f1ebc098630 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): my global_id is 124167 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc08dd90 con 0x7f1ebc098630 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1671224496 0 0) 0x7f1ea8004f10 con 0x7f1ebc0961d0 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ebc06fac0 con 0x7f1ebc0961d0 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1eac004830 con 0x7f1ebc0961d0 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): dump: 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.767 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318942224 0 0) 0x7f1ebc09c050 con 0x7f1ebc135530 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): my global_id is 114247 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8004f10 con 0x7f1ebc135530 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1885881305 0 0) 0x7f1ebc08dd90 con 0x7f1ebc098630 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ebc09c050 con 0x7f1ebc098630 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1eb0001ca0 con 0x7f1ebc098630 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient(hunting): dump: 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.769 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 820823473 0 0) 0x7f1ebc06fac0 con 0x7f1ebc0961d0 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.129:6789/0 conn(0x7f1ebc135530 legacy=0x7f1ebc147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6789/0 conn(0x7f1ebc098630 legacy=0x7f1ebc08d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 monclient: found mon.c 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc09d6c0 con 0x7f1ebc0961d0 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 5 monclient: authenticate success, global_id 104336 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 10 monclient: _renew_subs 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc08e110 con 0x7f1ebc0961d0 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 10 client.104336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 10 monclient: _renew_subs 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc08e590 con 0x7f1ebc0961d0 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 1 librados: init done 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eac004f40 con 0x7f1ebc0961d0 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1eac005250 con 0x7f1ebc0961d0 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1eb97fa700 4 set_mon_vals no callback set 2024-09-18T11:56:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 monclient: dump: 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.773 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1e9c006590 con 0x7f1e9c002840 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f1eac00fad0 con 0x7f1ebc0961d0 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(458..458 src has 1..458) v4 ==== 5402+0+0 (unknown 3936515613 0 0) 0x7f1eac043f90 con 0x7f1ebc0961d0 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 10 client.104336.objecter ms_dispatch 0x7f1ebc000fd0 osd_map(458..458 src has 1..458) v4 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 3 client.104336.objecter handle_osd_map got epochs [458,458] > 0 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 3 client.104336.objecter handle_osd_map decoding full epoch 458 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ea2ffd700 20 client.104336.objecter dump_active .. 0 homeless 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.752+0000 7f1ebb7fe700 10 monclient: get_auth_request con 0x7f1ea80385a0 auth_method 0 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1eba7fc700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6789/0 conn(0x7f1e9c002840 legacy=0x7f1e9c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 client.104336.objecter ms_handle_connect 0x7f1e9c002840 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 client.104336.objecter resend_mon_ops 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129802524 0 0) 0x7f1e9c006590 con 0x7f1e9c002840 2024-09-18T11:56:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 monclient(hunting): my global_id is 124170 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc06fac0 con 0x7f1e9c002840 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4199787253 0 0) 0x7f1ebc06fac0 con 0x7f1e9c002840 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e9c006590 con 0x7f1e9c002840 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1eb0002350 con 0x7f1e9c002840 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 monclient: dump: 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3681320626 0 0) 0x7f1e9c006590 con 0x7f1e9c002840 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e9c00a140 con 0x7f1e9c002840 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 client.104336.objecter ms_handle_connect 0x7f1ea80385a0 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f1eb0005810 con 0x7f1e9c002840 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:23.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6789/0 conn(0x7f1e9c002840 legacy=0x7f1e9c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.756+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6789/0 conn(0x7f1e9c002840 legacy=0x7f1e9c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:23.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.768+0000 7f1ebbfff700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.768+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1e9c007540 con 0x7f1e9c0068f0 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1eba7fc700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6789/0 conn(0x7f1e9c0068f0 legacy=0x7f1e9c006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 client.104336.objecter ms_handle_connect 0x7f1e9c0068f0 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 client.104336.objecter resend_mon_ops 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138648455 0 0) 0x7f1e9c007540 con 0x7f1e9c0068f0 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 monclient(hunting): my global_id is 124173 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e9c006590 con 0x7f1e9c0068f0 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1418718713 0 0) 0x7f1e9c006590 con 0x7f1e9c0068f0 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e9c007540 con 0x7f1e9c0068f0 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1eb0002930 con 0x7f1e9c0068f0 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 monclient: dump: 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 428934234 0 0) 0x7f1e9c007540 con 0x7f1e9c0068f0 2024-09-18T11:56:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7f1e9c0087b0 con 0x7f1e9c0068f0 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7f1eb0002cb0 con 0x7f1e9c0068f0 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6789/0 conn(0x7f1e9c0068f0 legacy=0x7f1e9c006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ea2ffd700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6789/0 conn(0x7f1e9c0068f0 legacy=0x7f1e9c006d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stdout: "ret": 0, 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-18T11:56:23.781 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:23.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 10 librados: watch_flush enter 2024-09-18T11:56:23.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 10 librados: watch_flush exit 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 20 client.104336.objecter shutdown clearing up homeless session... 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 10 client.104336.objecter successfully canceled tick 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.129:6812/20806 conn(0x7f1ea80385a0 legacy=0x7f1ea803aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 10 monclient: shutdown 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebb7fe700 1 -- 172.21.15.112:0/2664651238 reap_dead start 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 >> v1:172.21.15.112:6790/0 conn(0x7f1ebc0961d0 legacy=0x7f1ebc08cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 shutdown_connections 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 >> 172.21.15.112:0/2664651238 conn(0x7f1ebc0a2190 msgr2=0x7f1ebc09abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 shutdown_connections 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 1 -- 172.21.15.112:0/2664651238 wait complete. 2024-09-18T11:56:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.772+0000 7f1ebbfff700 1 librados: shutdown 2024-09-18T11:56:23.795 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T11:56:23.795 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T11:56:23.795 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:56:23.811 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T11:56:23.811 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T11:56:23.811 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.956+0000 7fe0d359e700 10 monclient: get_monmap_and_config 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.956+0000 7fe0d359e700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.956+0000 7fe0d359e700 10 monclient: monmap: 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:23.963818+0000 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:23.963818+0000 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.963 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.964 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:23.964 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:23.964 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 1 Processor -- start 2024-09-18T11:56:23.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 1 -- start start 2024-09-18T11:56:23.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 10 monclient: init 2024-09-18T11:56:23.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:56:23.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 10 monclient(hunting): picked mon.noname-a con 0x7fe0d40a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 10 monclient(hunting): picked mon.noname-b con 0x7fe0d409e370 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 10 monclient(hunting): picked mon.noname-c con 0x7fe0d4134740 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d259c700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe0d40a2040 legacy=0x7fe0d412e860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:23.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d406f150 con 0x7fe0d40a2040 2024-09-18T11:56:23.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d4096de0 con 0x7fe0d4134740 2024-09-18T11:56:23.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d406ee70 con 0x7fe0d409e370 2024-09-18T11:56:23.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d359e700 10 monclient(hunting): authenticate will time out at 1555.805666s 2024-09-18T11:56:23.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d1d9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe0d409e370 0x7fe0d409e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37938/0 (socket says 172.21.15.112:37938) 2024-09-18T11:56:23.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.960+0000 7fe0d1d9b700 1 -- 172.21.15.112:0/353261083 learned_addr learned my addr 172.21.15.112:0/353261083 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d1d9b700 1 -- 172.21.15.112:0/353261083 >> v1:172.21.15.112:6790/0 conn(0x7fe0d409e370 legacy=0x7fe0d409e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d2d9d700 1 -- 172.21.15.112:0/353261083 >> v1:172.21.15.129:6789/0 conn(0x7fe0d4134740 legacy=0x7fe0d4136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810390774 0 0) 0x7fe0d406ee70 con 0x7fe0d409e370 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient(hunting): my global_id is 104339 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0bc003570 con 0x7fe0d409e370 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746311836 0 0) 0x7fe0d4096de0 con 0x7fe0d4134740 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient(hunting): my global_id is 114250 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d406ee70 con 0x7fe0d4134740 2024-09-18T11:56:23.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3709325225 0 0) 0x7fe0bc003570 con 0x7fe0d409e370 2024-09-18T11:56:23.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 >> v1:172.21.15.129:6789/0 conn(0x7fe0d4134740 legacy=0x7fe0d4136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 >> v1:172.21.15.112:6789/0 conn(0x7fe0d40a2040 legacy=0x7fe0d412e860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:56:23.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 monclient: found mon.noname-b 2024-09-18T11:56:23.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d4096b00 con 0x7fe0d409e370 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 5 monclient: authenticate success, global_id 104339 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0c4001940 con 0x7fe0d409e370 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: dump: 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.972 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.973 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.973 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.973 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.973 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.973 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.973 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.973 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0c4001bd0 con 0x7fe0d409e370 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 4 set_mon_vals no callback set 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 1 -- 172.21.15.112:0/353261083 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0c4008ea0 con 0x7fe0d409e370 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d159a700 10 monclient: dump: 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.974 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 10 monclient: shutdown 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 -- 172.21.15.112:0/353261083 >> v1:172.21.15.112:6790/0 conn(0x7fe0d409e370 legacy=0x7fe0d409e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 -- 172.21.15.112:0/353261083 shutdown_connections 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 -- 172.21.15.112:0/353261083 >> 172.21.15.112:0/353261083 conn(0x7fe0d4067da0 msgr2=0x7fe0d40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:23.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 -- 172.21.15.112:0/353261083 shutdown_connections 2024-09-18T11:56:23.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 -- 172.21.15.112:0/353261083 wait complete. 2024-09-18T11:56:23.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 10 monclient: build_initial_monmap 2024-09-18T11:56:23.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 10 monclient: monmap: 2024-09-18T11:56:23.981 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:23.981 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:23.981 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:23.981 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:23.981 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 librados: starting msgr at 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 librados: starting objecter 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 Processor -- start 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 -- start start 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 librados: setting wanted keys 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 1 librados: calling monclient init 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.964+0000 7fe0d359e700 10 monclient: init 2024-09-18T11:56:23.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient(hunting): picked mon.noname-b con 0x7fe0d409e370 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient(hunting): picked mon.noname-a con 0x7fe0d40a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient(hunting): picked mon.noname-c con 0x7fe0d4134740 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d41304d0 con 0x7fe0d40a2040 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d412ac50 con 0x7fe0d409e370 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d4133680 con 0x7fe0d4134740 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d1d9b700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe0d40a2040 legacy=0x7fe0d412a540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient(hunting): authenticate will time out at 1555.812178s 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d2d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe0d4134740 0x7fe0d4145bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37942/0 (socket says 172.21.15.112:37942) 2024-09-18T11:56:23.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d2d9d700 1 -- 172.21.15.112:0/4140285627 learned_addr learned my addr 172.21.15.112:0/4140285627 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d2d9d700 1 -- 172.21.15.112:0/4140285627 >> v1:172.21.15.112:6790/0 conn(0x7fe0d4134740 legacy=0x7fe0d4145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 client.?.objecter ms_handle_connect 0x7fe0d4134740 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 client.?.objecter ms_handle_connect 0x7fe0d409e370 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d259c700 1 -- 172.21.15.112:0/4140285627 >> v1:172.21.15.129:6789/0 conn(0x7fe0d409e370 legacy=0x7fe0d4129e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712821249 0 0) 0x7fe0d4133680 con 0x7fe0d4134740 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): my global_id is 104342 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b4004f10 con 0x7fe0d4134740 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988037576 0 0) 0x7fe0d412ac50 con 0x7fe0d409e370 2024-09-18T11:56:23.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): my global_id is 114253 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d4133680 con 0x7fe0d409e370 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1470836038 0 0) 0x7fe0b4004f10 con 0x7fe0d4134740 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0d412ac50 con 0x7fe0d4134740 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0cc002000 con 0x7fe0d4134740 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): dump: 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.985 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 79951765 0 0) 0x7fe0d412ac50 con 0x7fe0d4134740 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 >> v1:172.21.15.129:6789/0 conn(0x7fe0d409e370 legacy=0x7fe0d4129e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 >> v1:172.21.15.112:6789/0 conn(0x7fe0d40a2040 legacy=0x7fe0d412a540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient: _finish_hunting 0 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 monclient: found mon.c 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d412f460 con 0x7fe0d4134740 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient: _finish_auth 0 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 5 monclient: authenticate success, global_id 104342 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient: _renew_subs 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d41307e0 con 0x7fe0d4134740 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 client.104342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0cc002630 con 0x7fe0d4134740 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe0cc0029c0 con 0x7fe0d4134740 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0c27fc700 10 monclient: dump: 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d0d99700 4 set_mon_vals no callback set 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient: _renew_subs 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d412b880 con 0x7fe0d4134740 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.968+0000 7fe0d359e700 1 librados: init done 2024-09-18T11:56:23.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe0cc002b20 con 0x7fe0d4134740 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0d359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0d359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0d359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d405f2e0 con 0x7fe0d4134740 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(458..458 src has 1..458) v4 ==== 5402+0+0 (unknown 3936515613 0 0) 0x7fe0cc044220 con 0x7fe0d4134740 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0c27fc700 10 client.104342.objecter ms_dispatch 0x7fe0d4000fd0 osd_map(458..458 src has 1..458) v4 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0c27fc700 3 client.104342.objecter handle_osd_map got epochs [458,458] > 0 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0c27fc700 3 client.104342.objecter handle_osd_map decoding full epoch 458 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0c27fc700 20 client.104342.objecter dump_active .. 0 homeless 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.972+0000 7fe0d259c700 10 monclient: get_auth_request con 0x7fe0b4038020 auth_method 0 2024-09-18T11:56:23.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.976+0000 7fe0c27fc700 10 client.104342.objecter ms_handle_connect 0x7fe0b4038020 2024-09-18T11:56:23.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.980+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fe0cc014e10 con 0x7fe0d4134740 2024-09-18T11:56:23.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.980+0000 7fe0c27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:23.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:23.980+0000 7fe0c27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:56:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe0d406df40 con 0x7fe0d4134740 2024-09-18T11:56:24.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0c27fc700 1 -- 172.21.15.112:0/4140285627 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 719605501) 0x7fe0cc014810 con 0x7fe0d4134740 2024-09-18T11:56:24.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0c27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:24.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0c27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:24.210 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:56:24.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 10 librados: watch_flush enter 2024-09-18T11:56:24.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 10 librados: watch_flush exit 2024-09-18T11:56:24.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 20 client.104342.objecter shutdown clearing up homeless session... 2024-09-18T11:56:24.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 10 client.104342.objecter successfully canceled tick 2024-09-18T11:56:24.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 >> v1:172.21.15.129:6812/20806 conn(0x7fe0b4038020 legacy=0x7fe0b403a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:24.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 10 monclient: shutdown 2024-09-18T11:56:24.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:24.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 >> v1:172.21.15.112:6790/0 conn(0x7fe0d4134740 legacy=0x7fe0d4145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:24.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 shutdown_connections 2024-09-18T11:56:24.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 >> 172.21.15.112:0/4140285627 conn(0x7fe0d4067da0 msgr2=0x7fe0d4124f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:24.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 shutdown_connections 2024-09-18T11:56:24.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 1 -- 172.21.15.112:0/4140285627 wait complete. 2024-09-18T11:56:24.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:24.204+0000 7fe0d359e700 1 librados: shutdown 2024-09-18T11:56:27.223 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:56:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient: get_monmap_and_config 2024-09-18T11:56:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient: build_initial_monmap 2024-09-18T11:56:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient: monmap: 2024-09-18T11:56:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:27.340803+0000 2024-09-18T11:56:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:27.340803+0000 2024-09-18T11:56:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:27.341 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:27.341 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:27.341 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:27.341 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:27.341 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 1 Processor -- start 2024-09-18T11:56:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 1 -- start start 2024-09-18T11:56:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient: init 2024-09-18T11:56:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fc39c0a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fc39c06c280 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:27.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fc39c06c670 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c06e950 con 0x7fc39c06c670 2024-09-18T11:56:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c0a2270 con 0x7fc39c0a1ea0 2024-09-18T11:56:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c06d130 con 0x7fc39c06c280 2024-09-18T11:56:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39bfff700 10 monclient(hunting): authenticate will time out at 1559.181671s 2024-09-18T11:56:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39b7fe700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fc39c06c670 legacy=0x7fc39c1388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39a7fc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc39c06c280 0x7fc39c135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37944/0 (socket says 172.21.15.112:37944) 2024-09-18T11:56:27.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.336+0000 7fc39a7fc700 1 -- 172.21.15.112:0/2103050759 learned_addr learned my addr 172.21.15.112:0/2103050759 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:27.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39a7fc700 1 -- 172.21.15.112:0/2103050759 >> v1:172.21.15.112:6790/0 conn(0x7fc39c06c280 legacy=0x7fc39c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:27.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023032490 0 0) 0x7fc39c06d130 con 0x7fc39c06c280 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient(hunting): my global_id is 104345 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c003570 con 0x7fc39c06c280 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39affd700 1 -- 172.21.15.112:0/2103050759 >> v1:172.21.15.129:6789/0 conn(0x7fc39c0a1ea0 legacy=0x7fc39c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1655880710 0 0) 0x7fc37c003570 con 0x7fc39c06c280 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 >> v1:172.21.15.129:6789/0 conn(0x7fc39c0a1ea0 legacy=0x7fc39c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 >> v1:172.21.15.112:6789/0 conn(0x7fc39c06c670 legacy=0x7fc39c1388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 monclient: found mon.noname-b 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:27.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c09e370 con 0x7fc39c06c280 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: _finish_auth 0 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc384001a00 con 0x7fc39c06c280 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: dump: 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:27.346 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 5 monclient: authenticate success, global_id 104345 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc384001b60 con 0x7fc39c06c280 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 4 set_mon_vals no callback set 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 1 -- 172.21.15.112:0/2103050759 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc384004a90 con 0x7fc39c06c280 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc399ffb700 10 monclient: dump: 2024-09-18T11:56:27.347 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:27.348 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 10 monclient: shutdown 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 -- 172.21.15.112:0/2103050759 >> v1:172.21.15.112:6790/0 conn(0x7fc39c06c280 legacy=0x7fc39c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 -- 172.21.15.112:0/2103050759 shutdown_connections 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 -- 172.21.15.112:0/2103050759 >> 172.21.15.112:0/2103050759 conn(0x7fc39c067da0 msgr2=0x7fc39c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 -- 172.21.15.112:0/2103050759 shutdown_connections 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 -- 172.21.15.112:0/2103050759 wait complete. 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 10 monclient: build_initial_monmap 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 10 monclient: monmap: 2024-09-18T11:56:27.353 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 librados: starting msgr at 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 librados: starting objecter 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 Processor -- start 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 -- start start 2024-09-18T11:56:27.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 librados: setting wanted keys 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 1 librados: calling monclient init 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.340+0000 7fc39bfff700 10 monclient: init 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fc39c06c280 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fc39c06c670 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fc39c0a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c0a1170 con 0x7fc39c0a1ea0 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c11edb0 con 0x7fc39c06c280 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c06e1a0 con 0x7fc39c06c670 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39bfff700 10 monclient(hunting): authenticate will time out at 1559.189140s 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39b7fe700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fc39c0a1ea0 legacy=0x7fc39c147ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:27.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39a7fc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc39c06c670 0x7fc39c144570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:37954/0 (socket says 172.21.15.112:37954) 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39a7fc700 1 -- 172.21.15.112:0/123409274 learned_addr learned my addr 172.21.15.112:0/123409274 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39a7fc700 1 -- 172.21.15.112:0/123409274 >> v1:172.21.15.112:6790/0 conn(0x7fc39c06c670 legacy=0x7fc39c144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 client.?.objecter ms_handle_connect 0x7fc39c06c670 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 client.?.objecter ms_handle_connect 0x7fc39c06c280 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc39affd700 1 -- 172.21.15.112:0/123409274 >> v1:172.21.15.129:6789/0 conn(0x7fc39c06c280 legacy=0x7fc39c13f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983904108 0 0) 0x7fc39c06e1a0 con 0x7fc39c06c670 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 monclient(hunting): my global_id is 104348 2024-09-18T11:56:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380004f10 con 0x7fc39c06c670 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735943859 0 0) 0x7fc39c11edb0 con 0x7fc39c06c280 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 10 monclient(hunting): my global_id is 114259 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.344+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c06e1a0 con 0x7fc39c06c280 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3510343664 0 0) 0x7fc380004f10 con 0x7fc39c06c670 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc39c11edb0 con 0x7fc39c06c670 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc3840019c0 con 0x7fc39c06c670 2024-09-18T11:56:27.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient(hunting): dump: 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:27.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284064152 0 0) 0x7fc39c06e1a0 con 0x7fc39c06c280 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc380004f10 con 0x7fc39c06c280 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc38c001c40 con 0x7fc39c06c280 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient(hunting): dump: 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:27.359 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2528979895 0 0) 0x7fc39c11edb0 con 0x7fc39c06c670 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 >> v1:172.21.15.129:6789/0 conn(0x7fc39c06c280 legacy=0x7fc39c13f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 >> v1:172.21.15.112:6789/0 conn(0x7fc39c0a1ea0 legacy=0x7fc39c147ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 monclient: found mon.c 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c120360 con 0x7fc39c06c670 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient: _finish_auth 0 2024-09-18T11:56:27.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 5 monclient: authenticate success, global_id 104348 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 10 monclient: _renew_subs 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39c11e8a0 con 0x7fc39c06c670 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 10 client.104348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 10 monclient: _renew_subs 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc39c1398a0 con 0x7fc39c06c670 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 1 librados: init done 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc384004fc0 con 0x7fc39c06c670 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc3840052d0 con 0x7fc39c06c670 2024-09-18T11:56:27.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc3997fa700 4 set_mon_vals no callback set 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 monclient: dump: 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:27.362 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fc38400fb50 con 0x7fc39c06c670 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3740050b0 con 0x7fc39c06c670 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(458..458 src has 1..458) v4 ==== 5402+0+0 (unknown 3936515613 0 0) 0x7fc384044160 con 0x7fc39c06c670 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 10 client.104348.objecter ms_dispatch 0x7fc39c000fd0 osd_map(458..458 src has 1..458) v4 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 3 client.104348.objecter handle_osd_map got epochs [458,458] > 0 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 3 client.104348.objecter handle_osd_map decoding full epoch 458 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc392ffd700 20 client.104348.objecter dump_active .. 0 homeless 2024-09-18T11:56:27.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.348+0000 7fc39affd700 10 monclient: get_auth_request con 0x7fc380038260 auth_method 0 2024-09-18T11:56:27.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.352+0000 7fc392ffd700 10 client.104348.objecter ms_handle_connect 0x7fc380038260 2024-09-18T11:56:27.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.360+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fc384014d80 con 0x7fc39c06c670 2024-09-18T11:56:27.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.360+0000 7fc392ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:27.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.360+0000 7fc392ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.580+0000 7fc39bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:56:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc3740058e0 con 0x7fc39c06c670 2024-09-18T11:56:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc392ffd700 1 -- 172.21.15.112:0/123409274 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1461603948) 0x7fc384014780 con 0x7fc39c06c670 2024-09-18T11:56:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc392ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:27.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc392ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:27.589 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:56:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 10 librados: watch_flush enter 2024-09-18T11:56:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 10 librados: watch_flush exit 2024-09-18T11:56:27.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 20 client.104348.objecter shutdown clearing up homeless session... 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 10 client.104348.objecter successfully canceled tick 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 >> v1:172.21.15.129:6812/20806 conn(0x7fc380038260 legacy=0x7fc38003a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 10 monclient: shutdown 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 >> v1:172.21.15.112:6790/0 conn(0x7fc39c06c670 legacy=0x7fc39c144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 shutdown_connections 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 >> 172.21.15.112:0/123409274 conn(0x7fc39c067da0 msgr2=0x7fc39c09fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 shutdown_connections 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 1 -- 172.21.15.112:0/123409274 wait complete. 2024-09-18T11:56:27.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:27.584+0000 7fc39bfff700 1 librados: shutdown 2024-09-18T11:56:30.602 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:56:30.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient: get_monmap_and_config 2024-09-18T11:56:30.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient: build_initial_monmap 2024-09-18T11:56:30.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient: monmap: 2024-09-18T11:56:30.714 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:30.714 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:30.714 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:30.714986+0000 2024-09-18T11:56:30.714 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:30.714986+0000 2024-09-18T11:56:30.714 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:30.715 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:30.715 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:30.715 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:30.715 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:30.715 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:30.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 1 Processor -- start 2024-09-18T11:56:30.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 1 -- start start 2024-09-18T11:56:30.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient: init 2024-09-18T11:56:30.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:30.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:30.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient(hunting): picked mon.noname-b con 0x7f8b340a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:30.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient(hunting): picked mon.noname-a con 0x7f8b3406c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient(hunting): picked mon.noname-c con 0x7f8b3406c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3406e950 con 0x7f8b3406c280 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340a2270 con 0x7f8b3406c670 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3406d130 con 0x7f8b340a1ea0 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b39d84700 10 monclient(hunting): authenticate will time out at 1562.555872s 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.712+0000 7f8b33fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b34135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.716+0000 7f8b38d82700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8b340a1ea0 0x7f8b3406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:51802/0 (socket says 172.21.15.112:51802) 2024-09-18T11:56:30.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.716+0000 7f8b38d82700 1 -- 172.21.15.112:0/3345298247 learned_addr learned my addr 172.21.15.112:0/3345298247 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:30.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.716+0000 7f8b38d82700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6790/0 conn(0x7f8b340a1ea0 legacy=0x7f8b3406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:30.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.716+0000 7f8b39583700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.129:6789/0 conn(0x7f8b3406c670 legacy=0x7f8b341388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:30.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:30.916+0000 7f8b33fff700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b34135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:31.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:31.320+0000 7f8b33fff700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b34135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:32.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:32.124+0000 7f8b33fff700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b34135170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:33.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.712+0000 7f8b327fc700 10 monclient(hunting): tick 2024-09-18T11:56:33.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.712+0000 7f8b327fc700 1 monclient(hunting): continuing hunt 2024-09-18T11:56:33.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.712+0000 7f8b327fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:56:33.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.712+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6790/0 conn(0x7f8b340a1ea0 legacy=0x7f8b3406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:33.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.129:6789/0 conn(0x7f8b3406c670 legacy=0x7f8b341388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b34135170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 10 monclient(hunting): picked mon.noname-a con 0x7f8b3406c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 10 monclient(hunting): picked mon.noname-c con 0x7f8b3406c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 10 monclient(hunting): picked mon.noname-b con 0x7f8b340a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3409e370 con 0x7f8b3406c280 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b38d82700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3406e950 con 0x7f8b3406c670 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b18010040 con 0x7f8b340a1ea0 2024-09-18T11:56:33.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b327fc700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:33.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b39583700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6790/0 conn(0x7f8b340a1ea0 legacy=0x7f8b1800c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:33.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.716+0000 7f8b33fff700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.129:6789/0 conn(0x7f8b3406c670 legacy=0x7f8b18008ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:33.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:33.920+0000 7f8b38d82700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:34.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:34.324+0000 7f8b38d82700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:35.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:35.128+0000 7f8b38d82700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 10 monclient(hunting): tick 2024-09-18T11:56:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 1 monclient(hunting): continuing hunt 2024-09-18T11:56:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:56:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6790/0 conn(0x7f8b340a1ea0 legacy=0x7f8b1800c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.129:6789/0 conn(0x7f8b3406c670 legacy=0x7f8b18008ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:56:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 10 monclient(hunting): picked mon.noname-b con 0x7f8b1800f0f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 10 monclient(hunting): picked mon.noname-a con 0x7f8b3406c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 10 monclient(hunting): picked mon.noname-c con 0x7f8b3406c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b18011300 con 0x7f8b3406c280 2024-09-18T11:56:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b33fff700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3409e370 con 0x7f8b3406c670 2024-09-18T11:56:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 1 -- 172.21.15.112:0/3345298247 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1800a790 con 0x7f8b1800f0f0 2024-09-18T11:56:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b327fc700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b38d82700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6790/0 conn(0x7f8b1800f0f0 legacy=0x7f8b1800f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b39583700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.129:6789/0 conn(0x7f8b3406c670 legacy=0x7f8b18008ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.716+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 DispatchQueue::entry inject delay of 0.002000 2024-09-18T11:56:36.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:36.920+0000 7f8b33fff700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:37.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:37.324+0000 7f8b33fff700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:38.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.128+0000 7f8b33fff700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:38.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898609742 0 0) 0x7f8b1800a790 con 0x7f8b1800f0f0 2024-09-18T11:56:38.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:38.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:38.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient(hunting): my global_id is 104351 2024-09-18T11:56:38.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b14003570 con 0x7f8b1800f0f0 2024-09-18T11:56:38.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 746358566 0 0) 0x7f8b14003570 con 0x7f8b1800f0f0 2024-09-18T11:56:38.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.129:6789/0 conn(0x7f8b3406c670 legacy=0x7f8b18008ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:38.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b18005840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: _finish_hunting 0 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 monclient: found mon.noname-b 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1800b930 con 0x7f8b1800f0f0 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: _finish_auth 0 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8b24001960 con 0x7f8b1800f0f0 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b39d84700 5 monclient: authenticate success, global_id 104351 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b39d84700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:38.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: dump: 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b39d84700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b24001bc0 con 0x7f8b1800f0f0 2024-09-18T11:56:38.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 1 -- 172.21.15.112:0/3345298247 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8b24004a90 con 0x7f8b1800f0f0 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b39d84700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b39d84700 4 set_mon_vals no callback set 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b337fe700 10 monclient: dump: 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:38.681 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:38.682 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:38.682 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:38.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b39d84700 10 monclient: shutdown 2024-09-18T11:56:38.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b39d84700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:38.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.676+0000 7f8b39d84700 1 -- 172.21.15.112:0/3345298247 >> v1:172.21.15.112:6790/0 conn(0x7f8b1800f0f0 legacy=0x7f8b1800f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:38.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 -- 172.21.15.112:0/3345298247 shutdown_connections 2024-09-18T11:56:38.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 -- 172.21.15.112:0/3345298247 >> 172.21.15.112:0/3345298247 conn(0x7f8b34067da0 msgr2=0x7f8b340a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:38.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 -- 172.21.15.112:0/3345298247 shutdown_connections 2024-09-18T11:56:38.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 -- 172.21.15.112:0/3345298247 wait complete. 2024-09-18T11:56:38.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 10 monclient: build_initial_monmap 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 10 monclient: monmap: 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 librados: starting msgr at 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 librados: starting objecter 2024-09-18T11:56:38.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 Processor -- start 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 -- start start 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 librados: setting wanted keys 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 1 librados: calling monclient init 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 10 monclient: init 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.680+0000 7f8b39d84700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39d84700 10 monclient(hunting): picked mon.noname-c con 0x7f8b1800f0f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39d84700 10 monclient(hunting): picked mon.noname-a con 0x7f8b3406c280 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39d84700 10 monclient(hunting): picked mon.noname-b con 0x7f8b3406c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39d84700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b341423d0 con 0x7f8b3406c280 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39d84700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b34121b30 con 0x7f8b3406c670 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b33fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b34132150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39d84700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b34121850 con 0x7f8b1800f0f0 2024-09-18T11:56:38.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39d84700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:38.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39d84700 10 monclient(hunting): authenticate will time out at 1570.524307s 2024-09-18T11:56:38.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b38d82700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8b1800f0f0 0x7f8b341414f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:51824/0 (socket says 172.21.15.112:51824) 2024-09-18T11:56:38.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b38d82700 1 -- 172.21.15.112:0/3593233404 learned_addr learned my addr 172.21.15.112:0/3593233404 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:38.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b38d82700 1 -- 172.21.15.112:0/3593233404 >> v1:172.21.15.112:6790/0 conn(0x7f8b1800f0f0 legacy=0x7f8b341414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:38.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b30ff9700 10 client.?.objecter ms_handle_connect 0x7f8b1800f0f0 2024-09-18T11:56:38.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b30ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:38.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b30ff9700 10 client.?.objecter ms_handle_connect 0x7f8b3406c670 2024-09-18T11:56:38.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b30ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:38.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.684+0000 7f8b39583700 1 -- 172.21.15.112:0/3593233404 >> v1:172.21.15.129:6789/0 conn(0x7f8b3406c670 legacy=0x7f8b34149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:38.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.688+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513584275 0 0) 0x7f8b34121b30 con 0x7f8b3406c670 2024-09-18T11:56:38.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.688+0000 7f8b30ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:38.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.688+0000 7f8b30ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:38.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.688+0000 7f8b30ff9700 10 monclient(hunting): my global_id is 114262 2024-09-18T11:56:38.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.688+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1800feb0 con 0x7f8b3406c670 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3608708554 0 0) 0x7f8b1800feb0 con 0x7f8b3406c670 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b34121b30 con 0x7f8b3406c670 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8b2c001e40 con 0x7f8b3406c670 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient(hunting): dump: 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:38.695 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683173688 0 0) 0x7f8b34121850 con 0x7f8b1800f0f0 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient(hunting): my global_id is 104354 2024-09-18T11:56:38.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1800feb0 con 0x7f8b1800f0f0 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1922096941 0 0) 0x7f8b34121b30 con 0x7f8b3406c670 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 >> v1:172.21.15.112:6790/0 conn(0x7f8b1800f0f0 legacy=0x7f8b341414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 >> v1:172.21.15.112:6789/0 conn(0x7f8b3406c280 legacy=0x7f8b34132150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 monclient: found mon.b 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b341203a0 con 0x7f8b3406c670 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient: _finish_auth 0 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 5 monclient: authenticate success, global_id 114262 2024-09-18T11:56:38.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 10 monclient: _renew_subs 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b34121290 con 0x7f8b3406c670 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 10 client.114262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 10 monclient: _renew_subs 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b34132dd0 con 0x7f8b3406c670 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b39d84700 1 librados: init done 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b2c002530 con 0x7f8b3406c670 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8b2c0028c0 con 0x7f8b3406c670 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b32ffd700 4 set_mon_vals no callback set 2024-09-18T11:56:38.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.692+0000 7f8b30ff9700 10 monclient: dump: 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:38.699 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:38.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8b2c002a20 con 0x7f8b3406c670 2024-09-18T11:56:38.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b39d84700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b39d84700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b39d84700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c0050b0 con 0x7f8b3406c670 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(458..458 src has 1..458) v4 ==== 5402+0+0 (unknown 3936515613 0 0) 0x7f8b2c043f60 con 0x7f8b3406c670 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b30ff9700 10 client.114262.objecter ms_dispatch 0x7f8b34000fd0 osd_map(458..458 src has 1..458) v4 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b30ff9700 3 client.114262.objecter handle_osd_map got epochs [458,458] > 0 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b30ff9700 3 client.114262.objecter handle_osd_map decoding full epoch 458 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b30ff9700 20 client.114262.objecter dump_active .. 0 homeless 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.696+0000 7f8b38d82700 10 monclient: get_auth_request con 0x7f8b18038b60 auth_method 0 2024-09-18T11:56:38.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.700+0000 7f8b30ff9700 10 client.114262.objecter ms_handle_connect 0x7f8b18038b60 2024-09-18T11:56:38.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.700+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f8b2c002cd0 con 0x7f8b3406c670 2024-09-18T11:56:38.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.700+0000 7f8b30ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:38.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.700+0000 7f8b30ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:38.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.932+0000 7f8b39d84700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:56:38.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.932+0000 7f8b39d84700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:38.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.932+0000 7f8b39d84700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T11:56:38.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.932+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8b0c0058e0 con 0x7f8b3406c670 2024-09-18T11:56:38.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b30ff9700 1 -- 172.21.15.112:0/3593233404 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3568177973) 0x7f8b2c014520 con 0x7f8b3406c670 2024-09-18T11:56:38.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b30ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:56:38.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b30ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:38.936 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":56,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:56:38.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 10 librados: watch_flush enter 2024-09-18T11:56:38.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 10 librados: watch_flush exit 2024-09-18T11:56:38.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 20 client.114262.objecter shutdown clearing up homeless session... 2024-09-18T11:56:38.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 10 client.114262.objecter successfully canceled tick 2024-09-18T11:56:38.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 >> v1:172.21.15.129:6812/20806 conn(0x7f8b18038b60 legacy=0x7f8b1803b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:38.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 10 monclient: shutdown 2024-09-18T11:56:38.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:38.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 >> v1:172.21.15.129:6789/0 conn(0x7f8b3406c670 legacy=0x7f8b34149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:38.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 shutdown_connections 2024-09-18T11:56:38.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 >> 172.21.15.112:0/3593233404 conn(0x7f8b34067da0 msgr2=0x7f8b34133090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:38.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 shutdown_connections 2024-09-18T11:56:38.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 1 -- 172.21.15.112:0/3593233404 wait complete. 2024-09-18T11:56:38.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:38.936+0000 7f8b39d84700 1 librados: shutdown 2024-09-18T11:56:38.950 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T11:56:38.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T11:56:38.950 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:56:39.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient: get_monmap_and_config 2024-09-18T11:56:39.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient: build_initial_monmap 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient: monmap: 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:39.058401+0000 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:39.058401+0000 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:39.058 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 1 Processor -- start 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 1 -- start start 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient: init 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient(hunting): picked mon.noname-b con 0x7f3c780a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient(hunting): picked mon.noname-a con 0x7f3c7806e9d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient(hunting): picked mon.noname-c con 0x7f3c78135300 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c780a1600 con 0x7f3c7806e9d0 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7806eda0 con 0x7f3c78135300 2024-09-18T11:56:39.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7806b5a0 con 0x7f3c780a1230 2024-09-18T11:56:39.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:39.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c7d2e1700 10 monclient(hunting): authenticate will time out at 1570.899491s 2024-09-18T11:56:39.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c767fc700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f3c7806e9d0 legacy=0x7f3c7806ae90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:39.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c76ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3c780a1230 0x7f3c7806a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:51830/0 (socket says 172.21.15.112:51830) 2024-09-18T11:56:39.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.056+0000 7f3c76ffd700 1 -- 172.21.15.112:0/3104327685 learned_addr learned my addr 172.21.15.112:0/3104327685 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c76ffd700 1 -- 172.21.15.112:0/3104327685 >> v1:172.21.15.112:6790/0 conn(0x7f3c780a1230 legacy=0x7f3c7806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255676271 0 0) 0x7f3c7806b5a0 con 0x7f3c780a1230 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient(hunting): my global_id is 104357 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c60003570 con 0x7f3c780a1230 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c777fe700 1 -- 172.21.15.112:0/3104327685 >> v1:172.21.15.129:6789/0 conn(0x7f3c78135300 legacy=0x7f3c781376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225603287 0 0) 0x7f3c60003570 con 0x7f3c780a1230 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 >> v1:172.21.15.129:6789/0 conn(0x7f3c78135300 legacy=0x7f3c781376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 >> v1:172.21.15.112:6789/0 conn(0x7f3c7806e9d0 legacy=0x7f3c7806ae90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:56:39.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 monclient: found mon.noname-b 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7809f9e0 con 0x7f3c780a1230 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: _finish_auth 0 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3c6c001940 con 0x7f3c780a1230 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 5 monclient: authenticate success, global_id 104357 2024-09-18T11:56:39.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: dump: 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c6c001c10 con 0x7f3c780a1230 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 1 -- 172.21.15.112:0/3104327685 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3c6c004a90 con 0x7f3c780a1230 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 4 set_mon_vals no callback set 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c75ffb700 10 monclient: dump: 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 10 monclient: shutdown 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/3104327685 >> v1:172.21.15.112:6790/0 conn(0x7f3c780a1230 legacy=0x7f3c7806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/3104327685 shutdown_connections 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/3104327685 >> 172.21.15.112:0/3104327685 conn(0x7f3c780680d0 msgr2=0x7f3c780a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/3104327685 shutdown_connections 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.060+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/3104327685 wait complete. 2024-09-18T11:56:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient: build_initial_monmap 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient: monmap: 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 librados: starting msgr at 2024-09-18T11:56:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 librados: starting objecter 2024-09-18T11:56:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 Processor -- start 2024-09-18T11:56:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 -- start start 2024-09-18T11:56:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 librados: setting wanted keys 2024-09-18T11:56:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 librados: calling monclient init 2024-09-18T11:56:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient: init 2024-09-18T11:56:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient(hunting): picked mon.noname-c con 0x7f3c7806e9d0 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient(hunting): picked mon.noname-b con 0x7f3c780a1230 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient(hunting): picked mon.noname-a con 0x7f3c78135300 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7812fa50 con 0x7f3c78135300 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7812fd30 con 0x7f3c780a1230 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c777fe700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f3c78135300 legacy=0x7f3c781476b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c78130010 con 0x7f3c7806e9d0 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c7d2e1700 10 monclient(hunting): authenticate will time out at 1570.905773s 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c76ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f3c7806e9d0 0x7f3c78134850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:51836/0 (socket says 172.21.15.112:51836) 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.064+0000 7f3c76ffd700 1 -- 172.21.15.112:0/2768765237 learned_addr learned my addr 172.21.15.112:0/2768765237 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c76ffd700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6790/0 conn(0x7f3c7806e9d0 legacy=0x7f3c78134850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 client.?.objecter ms_handle_connect 0x7f3c7806e9d0 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c767fc700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.129:6789/0 conn(0x7f3c780a1230 legacy=0x7f3c7813f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092941174 0 0) 0x7f3c78130010 con 0x7f3c7806e9d0 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): my global_id is 104360 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c64004f10 con 0x7f3c7806e9d0 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 client.?.objecter ms_handle_connect 0x7f3c780a1230 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827446803 0 0) 0x7f3c7812fd30 con 0x7f3c780a1230 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): my global_id is 114268 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c78130010 con 0x7f3c780a1230 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1249467449 0 0) 0x7f3c64004f10 con 0x7f3c7806e9d0 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c7812fd30 con 0x7f3c7806e9d0 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3c6c0049d0 con 0x7f3c7806e9d0 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): dump: 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3521390010 0 0) 0x7f3c7812fd30 con 0x7f3c7806e9d0 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.129:6789/0 conn(0x7f3c780a1230 legacy=0x7f3c7813f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6789/0 conn(0x7f3c78135300 legacy=0x7f3c781476b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient: _finish_hunting 0 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 monclient: found mon.c 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c781302f0 con 0x7f3c7806e9d0 2024-09-18T11:56:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient: _finish_auth 0 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 5 monclient: authenticate success, global_id 104360 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 10 monclient: _renew_subs 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c78134f60 con 0x7f3c7806e9d0 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 10 client.104360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 10 monclient: _renew_subs 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7813fbf0 con 0x7f3c7806e9d0 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 1 librados: init done 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c6c001320 con 0x7f3c7806e9d0 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3c6c0051e0 con 0x7f3c7806e9d0 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c757fa700 4 set_mon_vals no callback set 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 monclient: dump: 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f3c6c00fa60 con 0x7f3c7806e9d0 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3c58006590 con 0x7f3c58002840 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(459..459 src has 1..459) v4 ==== 5402+0+0 (unknown 1917701801 0 0) 0x7f3c6c0440d0 con 0x7f3c7806e9d0 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 10 client.104360.objecter ms_dispatch 0x7f3c78000fd0 osd_map(459..459 src has 1..459) v4 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 3 client.104360.objecter handle_osd_map got epochs [459,459] > 0 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 3 client.104360.objecter handle_osd_map decoding full epoch 459 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c5effd700 20 client.104360.objecter dump_active .. 0 homeless 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.068+0000 7f3c777fe700 10 monclient: get_auth_request con 0x7f3c640380c0 auth_method 0 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c767fc700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6790/0 conn(0x7f3c58002840 legacy=0x7f3c58004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 client.104360.objecter ms_handle_connect 0x7f3c58002840 2024-09-18T11:56:39.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 client.104360.objecter resend_mon_ops 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382719609 0 0) 0x7f3c58006590 con 0x7f3c58002840 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 monclient(hunting): my global_id is 104363 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c7812fd30 con 0x7f3c58002840 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2791962988 0 0) 0x7f3c7812fd30 con 0x7f3c58002840 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c58006590 con 0x7f3c58002840 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3c68004470 con 0x7f3c58002840 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 monclient: dump: 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 client.104360.objecter ms_handle_connect 0x7f3c640380c0 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3611918954 0 0) 0x7f3c58006590 con 0x7f3c58002840 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c5800a140 con 0x7f3c58002840 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f3c68006750 con 0x7f3c58002840 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6790/0 conn(0x7f3c58002840 legacy=0x7f3c58004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.072+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6790/0 conn(0x7f3c58002840 legacy=0x7f3c58004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c7d2e1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:56:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3c58007550 con 0x7f3c58006900 2024-09-18T11:56:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c767fc700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6790/0 conn(0x7f3c58006900 legacy=0x7f3c58006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c5effd700 10 client.104360.objecter ms_handle_connect 0x7f3c58006900 2024-09-18T11:56:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c5effd700 10 client.104360.objecter resend_mon_ops 2024-09-18T11:56:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637702502 0 0) 0x7f3c58007550 con 0x7f3c58006900 2024-09-18T11:56:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c5effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c5effd700 10 monclient(hunting): my global_id is 104366 2024-09-18T11:56:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.084+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c58006590 con 0x7f3c58006900 2024-09-18T11:56:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2105785611 0 0) 0x7f3c58006590 con 0x7f3c58006900 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c58007550 con 0x7f3c58006900 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3c68004c10 con 0x7f3c58006900 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 10 monclient: dump: 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2497250852 0 0) 0x7f3c58007550 con 0x7f3c58006900 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3c580087b0 con 0x7f3c58006900 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3990 (unknown 0 0 1674729762) 0x7f3c68006980 con 0x7f3c58006900 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6790/0 conn(0x7f3c58006900 legacy=0x7f3c58006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c5effd700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6790/0 conn(0x7f3c58006900 legacy=0x7f3c58006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:39.090 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 56, 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:39.091 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:39.092 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:56:39.093 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:39.094 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:56:39.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.096 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:39.097 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.098 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:56:39.099 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.100 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.101 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:56:39.102 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 10 librados: watch_flush enter 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 10 librados: watch_flush exit 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 20 client.104360.objecter shutdown clearing up homeless session... 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 10 client.104360.objecter successfully canceled tick 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.129:6812/20806 conn(0x7f3c640380c0 legacy=0x7f3c6403a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 10 monclient: shutdown 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 >> v1:172.21.15.112:6790/0 conn(0x7f3c7806e9d0 legacy=0x7f3c78134850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c777fe700 1 -- 172.21.15.112:0/2768765237 reap_dead start 2024-09-18T11:56:39.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 shutdown_connections 2024-09-18T11:56:39.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 >> 172.21.15.112:0/2768765237 conn(0x7f3c780680d0 msgr2=0x7f3c7809cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:39.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 shutdown_connections 2024-09-18T11:56:39.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 1 -- 172.21.15.112:0/2768765237 wait complete. 2024-09-18T11:56:39.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.088+0000 7f3c7d2e1700 1 librados: shutdown 2024-09-18T11:56:39.105 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:56:39.105 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T11:56:39.105 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:56:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient: get_monmap_and_config 2024-09-18T11:56:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient: build_initial_monmap 2024-09-18T11:56:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient: monmap: 2024-09-18T11:56:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:56:39.213957+0000 2024-09-18T11:56:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:56:39.213957+0000 2024-09-18T11:56:39.213 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:39.214 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.214 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:39.214 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:56:39.214 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:56:39.214 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 1 Processor -- start 2024-09-18T11:56:39.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 1 -- start start 2024-09-18T11:56:39.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient: init 2024-09-18T11:56:39.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient(hunting): picked mon.noname-b con 0x7fe8c00a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient(hunting): picked mon.noname-a con 0x7fe8c006e9d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient(hunting): picked mon.noname-c con 0x7fe8c0135300 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c00a1600 con 0x7fe8c006e9d0 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c006eda0 con 0x7fe8c0135300 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c006b5a0 con 0x7fe8c00a1230 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8c5b1a700 10 monclient(hunting): authenticate will time out at 1571.054968s 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8b6ffd700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe8c006e9d0 legacy=0x7fe8c006ae90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8bf7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe8c00a1230 0x7fe8c006a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42870/0 (socket says 172.21.15.112:42870) 2024-09-18T11:56:39.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.212+0000 7fe8bf7fe700 1 -- 172.21.15.112:0/3585875393 learned_addr learned my addr 172.21.15.112:0/3585875393 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8bf7fe700 1 -- 172.21.15.112:0/3585875393 >> v1:172.21.15.112:6790/0 conn(0x7fe8c00a1230 legacy=0x7fe8c006a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8bffff700 1 -- 172.21.15.112:0/3585875393 >> v1:172.21.15.129:6789/0 conn(0x7fe8c0135300 legacy=0x7fe8c01376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713009613 0 0) 0x7fe8c006b5a0 con 0x7fe8c00a1230 2024-09-18T11:56:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient(hunting): my global_id is 104369 2024-09-18T11:56:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ac003570 con 0x7fe8c00a1230 2024-09-18T11:56:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381629187 0 0) 0x7fe8c006eda0 con 0x7fe8c0135300 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient(hunting): my global_id is 114271 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c006b5a0 con 0x7fe8c0135300 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967877701 0 0) 0x7fe8ac003570 con 0x7fe8c00a1230 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 >> v1:172.21.15.129:6789/0 conn(0x7fe8c0135300 legacy=0x7fe8c01376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 >> v1:172.21.15.112:6789/0 conn(0x7fe8c006e9d0 legacy=0x7fe8c006ae90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: _finish_hunting 0 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 monclient: found mon.noname-b 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c009f9e0 con 0x7fe8c00a1230 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: _finish_auth 0 2024-09-18T11:56:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe8a8001940 con 0x7fe8c00a1230 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 5 monclient: authenticate success, global_id 104369 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: dump: 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8a8001bd0 con 0x7fe8c00a1230 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 10 monclient: get_monmap_and_config success 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 1 -- 172.21.15.112:0/3585875393 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe8a8004a90 con 0x7fe8c00a1230 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 4 set_mon_vals no callback set 2024-09-18T11:56:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8beffd700 10 monclient: dump: 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 10 monclient: shutdown 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/3585875393 >> v1:172.21.15.112:6790/0 conn(0x7fe8c00a1230 legacy=0x7fe8c006a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/3585875393 shutdown_connections 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/3585875393 >> 172.21.15.112:0/3585875393 conn(0x7fe8c00680d0 msgr2=0x7fe8c00a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/3585875393 shutdown_connections 2024-09-18T11:56:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.216+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/3585875393 wait complete. 2024-09-18T11:56:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient: build_initial_monmap 2024-09-18T11:56:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient: monmap: 2024-09-18T11:56:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:56:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 librados: starting msgr at 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 librados: starting objecter 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 Processor -- start 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 -- start start 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 librados: setting wanted keys 2024-09-18T11:56:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 librados: calling monclient init 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient: init 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient: _reopen_session rank -1 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient(hunting): picked mon.noname-b con 0x7fe8c006e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient(hunting): picked mon.noname-c con 0x7fe8c00a1230 addr v1:172.21.15.112:6790/0 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient(hunting): picked mon.noname-a con 0x7fe8c0135300 addr v1:172.21.15.112:6789/0 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c0142010 con 0x7fe8c0135300 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c0122680 con 0x7fe8c006e9d0 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c009ce90 con 0x7fe8c00a1230 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient(hunting): _renew_subs 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8c5b1a700 10 monclient(hunting): authenticate will time out at 1571.061768s 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8bffff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe8c0135300 legacy=0x7fe8c0149c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:56:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8b6ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe8c00a1230 0x7fe8c0134550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42886/0 (socket says 172.21.15.112:42886) 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.220+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/2411523581 learned_addr learned my addr 172.21.15.112:0/2411523581 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b6ffd700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.112:6790/0 conn(0x7fe8c00a1230 legacy=0x7fe8c0134550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 client.?.objecter ms_handle_connect 0x7fe8c00a1230 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215324058 0 0) 0x7fe8c009ce90 con 0x7fe8c00a1230 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): my global_id is 104372 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a0004f10 con 0x7fe8c00a1230 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 client.?.objecter ms_handle_connect 0x7fe8c006e9d0 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8bf7fe700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6789/0 conn(0x7fe8c006e9d0 legacy=0x7fe8c0141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839149784 0 0) 0x7fe8c0122680 con 0x7fe8c006e9d0 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): my global_id is 114274 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c009ce90 con 0x7fe8c006e9d0 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289302592 0 0) 0x7fe8a0004f10 con 0x7fe8c00a1230 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8c0122680 con 0x7fe8c00a1230 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe8b0004560 con 0x7fe8c00a1230 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): dump: 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 124587569 0 0) 0x7fe8c0122680 con 0x7fe8c00a1230 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6789/0 conn(0x7fe8c006e9d0 legacy=0x7fe8c0141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.112:6789/0 conn(0x7fe8c0135300 legacy=0x7fe8c0149c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient: _finish_hunting 0 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 monclient: found mon.c 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:56:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c0122960 con 0x7fe8c00a1230 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient: _finish_auth 0 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 5 monclient: authenticate success, global_id 104372 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 10 monclient: _renew_subs 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c0134c60 con 0x7fe8c00a1230 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 10 client.104372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 10 monclient: _renew_subs 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c0121930 con 0x7fe8c00a1230 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 1 librados: init done 2024-09-18T11:56:39.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8b00047f0 con 0x7fe8c00a1230 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe8b0004b80 con 0x7fe8c00a1230 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 monclient: dump: 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.237 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8bdffb700 4 set_mon_vals no callback set 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe8a4006590 con 0x7fe8a4002840 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe8b0004ce0 con 0x7fe8c00a1230 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(459..459 src has 1..459) v4 ==== 5402+0+0 (unknown 1917701801 0 0) 0x7fe8b0043f80 con 0x7fe8c00a1230 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 10 client.104372.objecter ms_dispatch 0x7fe8c0000fd0 osd_map(459..459 src has 1..459) v4 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 3 client.104372.objecter handle_osd_map got epochs [459,459] > 0 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 3 client.104372.objecter handle_osd_map decoding full epoch 459 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8b7fff700 20 client.104372.objecter dump_active .. 0 homeless 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.224+0000 7fe8bffff700 10 monclient: get_auth_request con 0x7fe8a0037f20 auth_method 0 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8bf7fe700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6789/0 conn(0x7fe8a4002840 legacy=0x7fe8a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 client.104372.objecter ms_handle_connect 0x7fe8a4002840 2024-09-18T11:56:39.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 client.104372.objecter resend_mon_ops 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582572265 0 0) 0x7fe8a4006590 con 0x7fe8a4002840 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 monclient(hunting): my global_id is 114277 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c0122680 con 0x7fe8a4002840 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4067998155 0 0) 0x7fe8c0122680 con 0x7fe8a4002840 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8a4006590 con 0x7fe8a4002840 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe8a8004f30 con 0x7fe8a4002840 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 monclient: dump: 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 client.104372.objecter ms_handle_connect 0x7fe8a0037f20 2024-09-18T11:56:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1302188604 0 0) 0x7fe8a4006590 con 0x7fe8a4002840 2024-09-18T11:56:39.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8a400a140 con 0x7fe8a4002840 2024-09-18T11:56:39.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe8a8005090 con 0x7fe8a4002840 2024-09-18T11:56:39.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:56:39.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:56:39.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6789/0 conn(0x7fe8a4002840 legacy=0x7fe8a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.228+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6789/0 conn(0x7fe8a4002840 legacy=0x7fe8a4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:39.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.240+0000 7fe8c5b1a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:56:39.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.240+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe8a4009f10 con 0x7fe8a4006900 2024-09-18T11:56:39.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8bf7fe700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6789/0 conn(0x7fe8a4006900 legacy=0x7fe8a4006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:56:39.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 10 client.104372.objecter ms_handle_connect 0x7fe8a4006900 2024-09-18T11:56:39.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 10 client.104372.objecter resend_mon_ops 2024-09-18T11:56:39.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176812773 0 0) 0x7fe8a4009f10 con 0x7fe8a4006900 2024-09-18T11:56:39.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 10 monclient(hunting): my global_id is 114280 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a4006590 con 0x7fe8a4006900 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1640289302 0 0) 0x7fe8a4006590 con 0x7fe8a4006900 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8a4009f10 con 0x7fe8a4006900 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe8a80056b0 con 0x7fe8a4006900 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 10 monclient: dump: 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:56:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:56:39.248 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:56:39.248 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:56:39.248 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:56:39.248 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:56:39.248 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:56:39.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1644125880 0 0) 0x7fe8a4009f10 con 0x7fe8a4006900 2024-09-18T11:56:39.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.244+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe8a4008760 con 0x7fe8a4006900 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4162 (unknown 0 0 268100345) 0x7fe8a8005a00 con 0x7fe8a4006900 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8b7fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8b7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6789/0 conn(0x7fe8a4006900 legacy=0x7fe8a4006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8b7fff700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6789/0 conn(0x7fe8a4006900 legacy=0x7fe8a4006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:56:39.249 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 56, 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:39.250 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:39.251 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:56:39.252 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:56:39.253 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.254 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:39.255 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.257 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.258 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:56:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:56:39.261 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 10 librados: watch_flush enter 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 10 librados: watch_flush exit 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 20 client.104372.objecter shutdown clearing up homeless session... 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 10 client.104372.objecter successfully canceled tick 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.129:6812/20806 conn(0x7fe8a0037f20 legacy=0x7fe8a003a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 10 monclient: shutdown 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 >> v1:172.21.15.112:6790/0 conn(0x7fe8c00a1230 legacy=0x7fe8c0134550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8bffff700 1 -- 172.21.15.112:0/2411523581 reap_dead start 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 shutdown_connections 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 >> 172.21.15.112:0/2411523581 conn(0x7fe8c00680d0 msgr2=0x7fe8c009e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:56:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 shutdown_connections 2024-09-18T11:56:39.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 1 -- 172.21.15.112:0/2411523581 wait complete. 2024-09-18T11:56:39.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:56:39.248+0000 7fe8c5b1a700 1 librados: shutdown 2024-09-18T11:56:39.264 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:56:39.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T11:58:09.265 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T11:58:09.265 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T11:58:09.266 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T11:58:09.266 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T11:58:09.269 INFO:tasks.ceph.mon.a:Started 2024-09-18T11:58:09.269 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:58:09.269 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.365+0000 7fe28193f700 10 monclient: get_monmap_and_config 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.365+0000 7fe28193f700 10 monclient: build_initial_monmap 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.365+0000 7fe28193f700 10 monclient: monmap: 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:09.371836+0000 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:09.371836+0000 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:09.371 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:09.372 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:09.372 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:09.372 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:09.372 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:09.365+0000 7fd9d188fa00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 492 -> 0 (sync_force? mon rebuild?) 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 1 Processor -- start 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 1 -- start start 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 10 monclient: init 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 10 monclient(hunting): picked mon.noname-a con 0x7fe27c067680 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 10 monclient(hunting): picked mon.noname-c con 0x7fe27c0985f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 10 monclient(hunting): picked mon.noname-b con 0x7fe27c0989e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c096b70 con 0x7fe27c067680 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c132d00 con 0x7fe27c0985f0 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c067a50 con 0x7fe27c0989e0 2024-09-18T11:58:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:09.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28193f700 10 monclient(hunting): authenticate will time out at 1661.212910s 2024-09-18T11:58:09.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28093d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fe27c067680 legacy=0x7fe27c1325f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T11:58:09.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28113e700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe27c0989e0 0x7fe27c13a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47568/0 (socket says 172.21.15.112:47568) 2024-09-18T11:58:09.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28113e700 1 -- 172.21.15.112:0/522101509 learned_addr learned my addr 172.21.15.112:0/522101509 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:09.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe28113e700 1 -- 172.21.15.112:0/522101509 >> v1:172.21.15.112:6790/0 conn(0x7fe27c0989e0 legacy=0x7fe27c13a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:09.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe27bfff700 1 -- 172.21.15.112:0/522101509 >> v1:172.21.15.129:6789/0 conn(0x7fe27c0985f0 legacy=0x7fe27c1371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362598206 0 0) 0x7fe27c067a50 con 0x7fe27c0989e0 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe27b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe27b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe27b7fe700 10 monclient(hunting): my global_id is 104399 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe268003570 con 0x7fe27c0989e0 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.369+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234224520 0 0) 0x7fe27c132d00 con 0x7fe27c0985f0 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient(hunting): my global_id is 114307 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe27c067a50 con 0x7fe27c0985f0 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3690318663 0 0) 0x7fe268003570 con 0x7fe27c0989e0 2024-09-18T11:58:09.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 >> v1:172.21.15.129:6789/0 conn(0x7fe27c0985f0 legacy=0x7fe27c1371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 >> v1:172.21.15.112:6789/0 conn(0x7fe27c067680 legacy=0x7fe27c1325f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 monclient: found mon.noname-b 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c096de0 con 0x7fe27c0989e0 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: _finish_auth 0 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 5 monclient: authenticate success, global_id 104399 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:09.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe274001cc0 con 0x7fe27c0989e0 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: dump: 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:09.382 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe274001f50 con 0x7fe27c0989e0 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 4 set_mon_vals no callback set 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 1 -- 172.21.15.112:0/522101509 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe274004e10 con 0x7fe27c0989e0 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27b7fe700 10 monclient: dump: 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:09.383 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient: shutdown 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- 172.21.15.112:0/522101509 >> v1:172.21.15.112:6790/0 conn(0x7fe27c0989e0 legacy=0x7fe27c13a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- 172.21.15.112:0/522101509 shutdown_connections 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- 172.21.15.112:0/522101509 >> 172.21.15.112:0/522101509 conn(0x7fe27c0a1ec0 msgr2=0x7fe27c0a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:09.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- 172.21.15.112:0/522101509 shutdown_connections 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- 172.21.15.112:0/522101509 wait complete. 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient: build_initial_monmap 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient: monmap: 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:09.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 librados: starting msgr at 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 librados: starting objecter 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 Processor -- start 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- start start 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 librados: setting wanted keys 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 librados: calling monclient init 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient: init 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient(hunting): picked mon.noname-b con 0x7fe27c067680 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient(hunting): picked mon.noname-c con 0x7fe27c0985f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient(hunting): picked mon.noname-a con 0x7fe27c0989e0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c13ba50 con 0x7fe27c0989e0 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c141c00 con 0x7fe27c067680 2024-09-18T11:58:09.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c12c1e0 con 0x7fe27c0985f0 2024-09-18T11:58:09.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27bfff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe27c0985f0 0x7fe27c12efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47592/0 (socket says 172.21.15.112:47592) 2024-09-18T11:58:09.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:09.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe27bfff700 1 -- 172.21.15.112:0/795438297 learned_addr learned my addr 172.21.15.112:0/795438297 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:09.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.373+0000 7fe28193f700 10 monclient(hunting): authenticate will time out at 1661.219508s 2024-09-18T11:58:09.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe27bfff700 1 -- 172.21.15.112:0/795438297 >> v1:172.21.15.112:6790/0 conn(0x7fe27c0985f0 legacy=0x7fe27c12efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 DispatchQueue::entry inject delay of 0.002000 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe28113e700 1 -- 172.21.15.112:0/795438297 >> v1:172.21.15.112:6789/0 conn(0x7fe27c0989e0 legacy=0x7fe27c149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe28093d700 1 -- 172.21.15.112:0/795438297 >> v1:172.21.15.129:6789/0 conn(0x7fe27c067680 legacy=0x7fe27c1414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 client.?.objecter ms_handle_connect 0x7fe27c0985f0 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 client.?.objecter ms_handle_connect 0x7fe27c0989e0 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 client.?.objecter ms_handle_connect 0x7fe27c067680 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417078361 0 0) 0x7fe27c12c1e0 con 0x7fe27c0985f0 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:09.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 monclient(hunting): my global_id is 104402 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe25c004f10 con 0x7fe27c0985f0 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452020778 0 0) 0x7fe27c141c00 con 0x7fe27c067680 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 10 monclient(hunting): my global_id is 114310 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.377+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe27c12c1e0 con 0x7fe27c067680 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477363241 0 0) 0x7fe25c004f10 con 0x7fe27c0985f0 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe27c141c00 con 0x7fe27c0985f0 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe26c001c60 con 0x7fe27c0985f0 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient(hunting): dump: 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:09.392 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2506389836 0 0) 0x7fe27c141c00 con 0x7fe27c0985f0 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 >> v1:172.21.15.129:6789/0 conn(0x7fe27c067680 legacy=0x7fe27c1414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 >> v1:172.21.15.112:6789/0 conn(0x7fe27c0989e0 legacy=0x7fe27c149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:09.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 monclient: found mon.c 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c12c4c0 con 0x7fe27c0985f0 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient: _finish_auth 0 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 5 monclient: authenticate success, global_id 104402 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 10 monclient: _renew_subs 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27c12a6b0 con 0x7fe27c0985f0 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 10 client.104402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 10 monclient: _renew_subs 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:09.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27c12b280 con 0x7fe27c0985f0 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 1 librados: init done 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe26c002300 con 0x7fe27c0985f0 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe26c002610 con 0x7fe27c0985f0 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe27affd700 4 set_mon_vals no callback set 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 monclient: dump: 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:09.395 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe26c002770 con 0x7fe27c0985f0 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2600050b0 con 0x7fe27c0985f0 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(534..534 src has 1..534) v4 ==== 5737+0+0 (unknown 1484240301 0 0) 0x7fe26c044010 con 0x7fe27c0985f0 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 10 client.104402.objecter ms_dispatch 0x7fe27c000fd0 osd_map(534..534 src has 1..534) v4 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 3 client.104402.objecter handle_osd_map got epochs [534,534] > 0 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 3 client.104402.objecter handle_osd_map decoding full epoch 534 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe278ff9700 20 client.104402.objecter dump_active .. 0 homeless 2024-09-18T11:58:09.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.381+0000 7fe28093d700 10 monclient: get_auth_request con 0x7fe25c0383d0 auth_method 0 2024-09-18T11:58:09.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.385+0000 7fe278ff9700 10 client.104402.objecter ms_handle_connect 0x7fe25c0383d0 2024-09-18T11:58:09.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.389+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fe26c014aa0 con 0x7fe27c0985f0 2024-09-18T11:58:09.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.389+0000 7fe278ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:09.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.389+0000 7fe278ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:09.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.613+0000 7fe28193f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:58:09.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.613+0000 7fe28193f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:09.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.613+0000 7fe28193f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:09.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:09.613+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe2600058e0 con 0x7fe27c0985f0 2024-09-18T11:58:10.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.009+0000 7fe278ff9700 1 -- 172.21.15.112:0/795438297 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 997597794) 0x7fe26c0144a0 con 0x7fe27c0985f0 2024-09-18T11:58:10.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.009+0000 7fe278ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:10.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.009+0000 7fe278ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:10.015 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 10 librados: watch_flush enter 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 10 librados: watch_flush exit 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 20 client.104402.objecter shutdown clearing up homeless session... 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 10 client.104402.objecter successfully canceled tick 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 >> v1:172.21.15.129:6812/20806 conn(0x7fe25c0383d0 legacy=0x7fe25c03a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 10 monclient: shutdown 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 >> v1:172.21.15.112:6790/0 conn(0x7fe27c0985f0 legacy=0x7fe27c12efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 shutdown_connections 2024-09-18T11:58:10.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 >> 172.21.15.112:0/795438297 conn(0x7fe27c0a1ec0 msgr2=0x7fe27c0a08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:10.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 shutdown_connections 2024-09-18T11:58:10.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 1 -- 172.21.15.112:0/795438297 wait complete. 2024-09-18T11:58:10.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.017+0000 7fe28193f700 1 librados: shutdown 2024-09-18T11:58:10.036 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:58:10.036 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:58:10.037 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:58:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.137+0000 7fcdd8900700 10 monclient: get_monmap_and_config 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.137+0000 7fcdd8900700 10 monclient: build_initial_monmap 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.137+0000 7fcdd8900700 10 monclient: monmap: 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:10.143637+0000 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:10.143637+0000 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:10.143 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 1 Processor -- start 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 1 -- start start 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 10 monclient: init 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 10 monclient(hunting): picked mon.noname-a con 0x7fcdd40a1230 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 10 monclient(hunting): picked mon.noname-c con 0x7fcdd406e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 10 monclient(hunting): picked mon.noname-b con 0x7fcdd4135300 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd40a1600 con 0x7fcdd40a1230 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd406eda0 con 0x7fcdd406e9d0 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd406b5a0 con 0x7fcdd4135300 2024-09-18T11:58:10.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:10.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd8900700 10 monclient(hunting): authenticate will time out at 1661.985123s 2024-09-18T11:58:10.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd259c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcdd40a1230 0x7fcdd406a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34974/0 (socket says 172.21.15.112:34974) 2024-09-18T11:58:10.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd2d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fcdd4135300 0x7fcdd41376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47606/0 (socket says 172.21.15.112:47606) 2024-09-18T11:58:10.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd259c700 1 -- 172.21.15.112:0/2345092627 learned_addr learned my addr 172.21.15.112:0/2345092627 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:10.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd259c700 1 -- 172.21.15.112:0/2345092627 >> v1:172.21.15.112:6789/0 conn(0x7fcdd40a1230 legacy=0x7fcdd406a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd2d9d700 1 -- 172.21.15.112:0/2345092627 >> v1:172.21.15.112:6790/0 conn(0x7fcdd4135300 legacy=0x7fcdd41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd1d9b700 1 -- 172.21.15.112:0/2345092627 >> v1:172.21.15.129:6789/0 conn(0x7fcdd406e9d0 legacy=0x7fcdd406ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508943643 0 0) 0x7fcdd40a1600 con 0x7fcdd40a1230 2024-09-18T11:58:10.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd159a700 10 monclient(hunting): my global_id is 134097 2024-09-18T11:58:10.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.141+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdbc003570 con 0x7fcdd40a1230 2024-09-18T11:58:10.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183498616 0 0) 0x7fcdd406b5a0 con 0x7fcdd4135300 2024-09-18T11:58:10.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient(hunting): my global_id is 104405 2024-09-18T11:58:10.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd40a1600 con 0x7fcdd4135300 2024-09-18T11:58:10.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1484657734 0 0) 0x7fcdbc003570 con 0x7fcdd40a1230 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 >> v1:172.21.15.112:6790/0 conn(0x7fcdd4135300 legacy=0x7fcdd41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 >> v1:172.21.15.129:6789/0 conn(0x7fcdd406e9d0 legacy=0x7fcdd406ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: _finish_hunting 0 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 monclient: found mon.noname-a 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd409f9e0 con 0x7fcdd40a1230 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: _finish_auth 0 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 5 monclient: authenticate success, global_id 134097 2024-09-18T11:58:10.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcdc4001940 con 0x7fcdd40a1230 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: dump: 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.152 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdc4001bd0 con 0x7fcdd40a1230 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 4 set_mon_vals no callback set 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 1 -- 172.21.15.112:0/2345092627 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcdc4004a90 con 0x7fcdd40a1230 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd159a700 10 monclient: dump: 2024-09-18T11:58:10.153 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 10 monclient: shutdown 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 -- 172.21.15.112:0/2345092627 >> v1:172.21.15.112:6789/0 conn(0x7fcdd40a1230 legacy=0x7fcdd406a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 -- 172.21.15.112:0/2345092627 shutdown_connections 2024-09-18T11:58:10.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 -- 172.21.15.112:0/2345092627 >> 172.21.15.112:0/2345092627 conn(0x7fcdd40680d0 msgr2=0x7fcdd40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 -- 172.21.15.112:0/2345092627 shutdown_connections 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 -- 172.21.15.112:0/2345092627 wait complete. 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 10 monclient: build_initial_monmap 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 10 monclient: monmap: 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:10.155 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 librados: starting msgr at 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 librados: starting objecter 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 Processor -- start 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 -- start start 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 librados: setting wanted keys 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 1 librados: calling monclient init 2024-09-18T11:58:10.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 10 monclient: init 2024-09-18T11:58:10.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:10.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:58:10.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.145+0000 7fcdd8900700 10 monclient(hunting): picked mon.noname-b con 0x7fcdd406e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:10.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 monclient(hunting): picked mon.noname-a con 0x7fcdd40a1230 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:10.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 monclient(hunting): picked mon.noname-c con 0x7fcdd4135300 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:10.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd412e260 con 0x7fcdd40a1230 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd412f770 con 0x7fcdd406e9d0 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd409a990 con 0x7fcdd4135300 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 monclient(hunting): authenticate will time out at 1661.992137s 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd1d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcdd40a1230 0x7fcdd409a280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34978/0 (socket says 172.21.15.112:34978) 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd2d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fcdd4135300 0x7fcdd414a1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47622/0 (socket says 172.21.15.112:47622) 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd1d9b700 1 -- 172.21.15.112:0/1577790414 learned_addr learned my addr 172.21.15.112:0/1577790414 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd2d9d700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6790/0 conn(0x7fcdd4135300 legacy=0x7fcdd414a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 client.?.objecter ms_handle_connect 0x7fcdd4135300 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 client.?.objecter ms_handle_connect 0x7fcdd40a1230 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd1d9b700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6789/0 conn(0x7fcdd40a1230 legacy=0x7fcdd409a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd259c700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.129:6789/0 conn(0x7fcdd406e9d0 legacy=0x7fcdd4099b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 client.?.objecter ms_handle_connect 0x7fcdd406e9d0 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618347109 0 0) 0x7fcdd409a990 con 0x7fcdd4135300 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): my global_id is 104408 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb4004f10 con 0x7fcdd4135300 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692553678 0 0) 0x7fcdd412e260 con 0x7fcdd40a1230 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): my global_id is 134100 2024-09-18T11:58:10.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd409a990 con 0x7fcdd40a1230 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1131973147 0 0) 0x7fcdb4004f10 con 0x7fcdd4135300 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdd412e260 con 0x7fcdd4135300 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcdcc001fb0 con 0x7fcdd4135300 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): dump: 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.164 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527635899 0 0) 0x7fcdd409a990 con 0x7fcdd40a1230 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdb4004f10 con 0x7fcdd40a1230 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcdc8001be0 con 0x7fcdd40a1230 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient(hunting): dump: 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.165 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2702087745 0 0) 0x7fcdd412e260 con 0x7fcdd4135300 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.129:6789/0 conn(0x7fcdd406e9d0 legacy=0x7fcdd4099b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6789/0 conn(0x7fcdd40a1230 legacy=0x7fcdd409a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient: _finish_hunting 0 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 monclient: found mon.c 2024-09-18T11:58:10.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd406ef90 con 0x7fcdd4135300 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 10 monclient: _finish_auth 0 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdc27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 5 monclient: authenticate success, global_id 104408 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 monclient: _renew_subs 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd412f070 con 0x7fcdd4135300 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 client.104408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 monclient: _renew_subs 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd409aed0 con 0x7fcdd4135300 2024-09-18T11:58:10.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.149+0000 7fcdd8900700 1 librados: init done 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdcc004e60 con 0x7fcdd4135300 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcdcc0051f0 con 0x7fcdd4135300 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdd0d99700 4 set_mon_vals no callback set 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 monclient: dump: 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.168 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdd8900700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcdb8006590 con 0x7fcdb8002840 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fcdcc005450 con 0x7fcdd4135300 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(534..534 src has 1..534) v4 ==== 5737+0+0 (unknown 1484240301 0 0) 0x7fcdcc044240 con 0x7fcdd4135300 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 client.104408.objecter ms_dispatch 0x7fcdd4000fd0 osd_map(534..534 src has 1..534) v4 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 3 client.104408.objecter handle_osd_map got epochs [534,534] > 0 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 3 client.104408.objecter handle_osd_map decoding full epoch 534 2024-09-18T11:58:10.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 20 client.104408.objecter dump_active .. 0 homeless 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdd1d9b700 10 monclient: get_auth_request con 0x7fcdb4038370 auth_method 0 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 client.104408.objecter ms_handle_connect 0x7fcdb8002840 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 client.104408.objecter resend_mon_ops 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdd259c700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6789/0 conn(0x7fcdb8002840 legacy=0x7fcdb8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744823563 0 0) 0x7fcdb8006590 con 0x7fcdb8002840 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 10 monclient(hunting): my global_id is 134103 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.153+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd412e260 con 0x7fcdb8002840 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1196750517 0 0) 0x7fcdd412e260 con 0x7fcdb8002840 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdb8006590 con 0x7fcdb8002840 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcdc4004aa0 con 0x7fcdb8002840 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:10.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 10 monclient: dump: 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3401014298 0 0) 0x7fcdb8006590 con 0x7fcdb8002840 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcdb800a140 con 0x7fcdb8002840 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 10 client.104408.objecter ms_handle_connect 0x7fcdb4038370 2024-09-18T11:58:10.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fcdc4001820 con 0x7fcdb8002840 2024-09-18T11:58:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6789/0 conn(0x7fcdb8002840 legacy=0x7fcdb8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.157+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6789/0 conn(0x7fcdb8002840 legacy=0x7fcdb8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdd8900700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:58:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcdb8009670 con 0x7fcdb80090e0 2024-09-18T11:58:10.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdd259c700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6789/0 conn(0x7fcdb80090e0 legacy=0x7fcdb8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 10 client.104408.objecter ms_handle_connect 0x7fcdb80090e0 2024-09-18T11:58:10.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 10 client.104408.objecter resend_mon_ops 2024-09-18T11:58:10.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925399203 0 0) 0x7fcdb8009670 con 0x7fcdb80090e0 2024-09-18T11:58:10.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 10 monclient(hunting): my global_id is 134106 2024-09-18T11:58:10.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb8006590 con 0x7fcdb80090e0 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3446966576 0 0) 0x7fcdb8006590 con 0x7fcdb80090e0 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdb8009670 con 0x7fcdb80090e0 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcdc4005390 con 0x7fcdb80090e0 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.169+0000 7fcdc27fc700 10 monclient: dump: 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.176 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.177 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.177 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.177 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2737729767 0 0) 0x7fcdb8009670 con 0x7fcdb80090e0 2024-09-18T11:58:10.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcdb8009930 con 0x7fcdb80090e0 2024-09-18T11:58:10.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 3386148629) 0x7fcdc4006f10 con 0x7fcdb80090e0 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdc27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdc27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6789/0 conn(0x7fcdb80090e0 legacy=0x7fcdb8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdc27fc700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6789/0 conn(0x7fcdb80090e0 legacy=0x7fcdb8008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:58:10.178 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 60, 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.179 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.180 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:58:10.181 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.182 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:10.183 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.184 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.185 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.186 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.187 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:58:10.189 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:10.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 10 librados: watch_flush enter 2024-09-18T11:58:10.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 10 librados: watch_flush exit 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 20 client.104408.objecter shutdown clearing up homeless session... 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 10 client.104408.objecter successfully canceled tick 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.129:6812/20806 conn(0x7fcdb4038370 legacy=0x7fcdb403a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 10 monclient: shutdown 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 >> v1:172.21.15.112:6790/0 conn(0x7fcdd4135300 legacy=0x7fcdd414a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd2d9d700 1 -- 172.21.15.112:0/1577790414 reap_dead start 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 shutdown_connections 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 >> 172.21.15.112:0/1577790414 conn(0x7fcdd40680d0 msgr2=0x7fcdd406e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 shutdown_connections 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 1 -- 172.21.15.112:0/1577790414 wait complete. 2024-09-18T11:58:10.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.173+0000 7fcdd8900700 1 librados: shutdown 2024-09-18T11:58:10.192 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:58:10.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:58:10.192 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient: get_monmap_and_config 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient: build_initial_monmap 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient: monmap: 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:10.304834+0000 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:10.304834+0000 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:10.304 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.305 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:10.305 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:10.305 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:10.305 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 1 Processor -- start 2024-09-18T11:58:10.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 1 -- start start 2024-09-18T11:58:10.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient: init 2024-09-18T11:58:10.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient(hunting): picked mon.noname-a con 0x7fe30006a8b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient(hunting): picked mon.noname-c con 0x7fe300132330 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient(hunting): picked mon.noname-b con 0x7fe300132e40 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe300096ad0 con 0x7fe30006a8b0 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000a20e0 con 0x7fe300132330 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000a1740 con 0x7fe300132e40 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe306d1c700 10 monclient(hunting): authenticate will time out at 1662.146990s 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe3052b9700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe300132e40 0x7fe30013a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47634/0 (socket says 172.21.15.112:47634) 2024-09-18T11:58:10.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.301+0000 7fe3052b9700 1 -- 172.21.15.112:0/3175162113 learned_addr learned my addr 172.21.15.112:0/3175162113 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:10.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe3052b9700 1 -- 172.21.15.112:0/3175162113 >> v1:172.21.15.112:6790/0 conn(0x7fe300132e40 legacy=0x7fe30013a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe304ab8700 1 -- 172.21.15.112:0/3175162113 >> v1:172.21.15.112:6789/0 conn(0x7fe30006a8b0 legacy=0x7fe300097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2fffff700 1 -- 172.21.15.112:0/3175162113 >> v1:172.21.15.129:6789/0 conn(0x7fe300132330 legacy=0x7fe300132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245439951 0 0) 0x7fe3000a1740 con 0x7fe300132e40 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient(hunting): my global_id is 104411 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e8003570 con 0x7fe300132e40 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865899725 0 0) 0x7fe300096ad0 con 0x7fe30006a8b0 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient(hunting): my global_id is 134109 2024-09-18T11:58:10.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3000a1740 con 0x7fe30006a8b0 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1581651515 0 0) 0x7fe2e8003570 con 0x7fe300132e40 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 >> v1:172.21.15.129:6789/0 conn(0x7fe300132330 legacy=0x7fe300132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 >> v1:172.21.15.112:6789/0 conn(0x7fe30006a8b0 legacy=0x7fe300097b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 monclient: found mon.noname-b 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30013ece0 con 0x7fe300132e40 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: _finish_auth 0 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe2f8001cc0 con 0x7fe300132e40 2024-09-18T11:58:10.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: dump: 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 5 monclient: authenticate success, global_id 104411 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2f8001f50 con 0x7fe300132e40 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 1 -- 172.21.15.112:0/3175162113 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe2f8004e10 con 0x7fe300132e40 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 4 set_mon_vals no callback set 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe2ff7fe700 10 monclient: dump: 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.316 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 10 monclient: shutdown 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 1 -- 172.21.15.112:0/3175162113 >> v1:172.21.15.112:6790/0 conn(0x7fe300132e40 legacy=0x7fe30013a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 1 -- 172.21.15.112:0/3175162113 shutdown_connections 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 1 -- 172.21.15.112:0/3175162113 >> 172.21.15.112:0/3175162113 conn(0x7fe3000a5b70 msgr2=0x7fe3000a5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:10.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 1 -- 172.21.15.112:0/3175162113 shutdown_connections 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 1 -- 172.21.15.112:0/3175162113 wait complete. 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 10 monclient: build_initial_monmap 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.305+0000 7fe306d1c700 10 monclient: monmap: 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:10.318 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 librados: starting msgr at 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 librados: starting objecter 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 Processor -- start 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 -- start start 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 librados: setting wanted keys 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 librados: calling monclient init 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 10 monclient: init 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 10 monclient(hunting): picked mon.noname-a con 0x7fe30006a8b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:10.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 10 monclient(hunting): picked mon.noname-b con 0x7fe300132330 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:10.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 10 monclient(hunting): picked mon.noname-c con 0x7fe300132e40 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:10.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000a4340 con 0x7fe30006a8b0 2024-09-18T11:58:10.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000a22e0 con 0x7fe300132330 2024-09-18T11:58:10.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe300133cb0 con 0x7fe300132e40 2024-09-18T11:58:10.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:10.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe306d1c700 10 monclient(hunting): authenticate will time out at 1662.152742s 2024-09-18T11:58:10.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe304ab8700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe30006a8b0 0x7fe300142040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35018/0 (socket says 172.21.15.112:35018) 2024-09-18T11:58:10.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe304ab8700 1 -- 172.21.15.112:0/221553651 learned_addr learned my addr 172.21.15.112:0/221553651 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:10.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe304ab8700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6789/0 conn(0x7fe30006a8b0 legacy=0x7fe300142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 client.?.objecter ms_handle_connect 0x7fe30006a8b0 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe3052b9700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6790/0 conn(0x7fe300132e40 legacy=0x7fe30014a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 client.?.objecter ms_handle_connect 0x7fe300132e40 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294435641 0 0) 0x7fe3000a4340 con 0x7fe30006a8b0 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): my global_id is 134112 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ec004f10 con 0x7fe30006a8b0 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 client.?.objecter ms_handle_connect 0x7fe300132330 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fffff700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.129:6789/0 conn(0x7fe300132330 legacy=0x7fe300142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.325 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178680193 0 0) 0x7fe300133cb0 con 0x7fe300132e40 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): my global_id is 104414 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3000a4340 con 0x7fe300132e40 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580729578 0 0) 0x7fe2ec004f10 con 0x7fe30006a8b0 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe300133cb0 con 0x7fe30006a8b0 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe2f0001ce0 con 0x7fe30006a8b0 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): dump: 2024-09-18T11:58:10.326 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1801921161 0 0) 0x7fe3000a4340 con 0x7fe300132e40 2024-09-18T11:58:10.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2ec004f10 con 0x7fe300132e40 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe2f8004bb0 con 0x7fe300132e40 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.309+0000 7fe2fcff9700 10 monclient(hunting): dump: 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.328 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1441236810 0 0) 0x7fe2ec004f10 con 0x7fe300132e40 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.129:6789/0 conn(0x7fe300132330 legacy=0x7fe300142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6789/0 conn(0x7fe30006a8b0 legacy=0x7fe300142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 10 monclient: _finish_hunting 0 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 monclient: found mon.c 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3001337a0 con 0x7fe300132e40 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 10 monclient: _finish_auth 0 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 5 monclient: authenticate success, global_id 104414 2024-09-18T11:58:10.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 10 monclient: _renew_subs 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe300134370 con 0x7fe300132e40 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 10 client.104414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 10 monclient: _renew_subs 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3001365b0 con 0x7fe300132e40 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 1 librados: init done 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2f8001900 con 0x7fe300132e40 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe2f8005110 con 0x7fe300132e40 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fe7fc700 4 set_mon_vals no callback set 2024-09-18T11:58:10.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 10 monclient: dump: 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe2e0006590 con 0x7fe2e0002840 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe2f800f990 con 0x7fe300132e40 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(534..534 src has 1..534) v4 ==== 5737+0+0 (unknown 1484240301 0 0) 0x7fe2f8044150 con 0x7fe300132e40 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 10 client.104414.objecter ms_dispatch 0x7fe300000fd0 osd_map(534..534 src has 1..534) v4 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 3 client.104414.objecter handle_osd_map got epochs [534,534] > 0 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 3 client.104414.objecter handle_osd_map decoding full epoch 534 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fcff9700 20 client.104414.objecter dump_active .. 0 homeless 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.313+0000 7fe2fffff700 10 monclient: get_auth_request con 0x7fe2ec038320 auth_method 0 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 client.104414.objecter ms_handle_connect 0x7fe2e0002840 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 client.104414.objecter resend_mon_ops 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe304ab8700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6790/0 conn(0x7fe2e0002840 legacy=0x7fe2e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261219951 0 0) 0x7fe2e0006590 con 0x7fe2e0002840 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 monclient(hunting): my global_id is 104417 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ec004f10 con 0x7fe2e0002840 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 637973951 0 0) 0x7fe2ec004f10 con 0x7fe2e0002840 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2e0006590 con 0x7fe2e0002840 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe2f0002710 con 0x7fe2e0002840 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 monclient: dump: 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.333 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2406625378 0 0) 0x7fe2e0006590 con 0x7fe2e0002840 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2e000a140 con 0x7fe2e0002840 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 client.104414.objecter ms_handle_connect 0x7fe2ec038320 2024-09-18T11:58:10.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe2f0002a30 con 0x7fe2e0002840 2024-09-18T11:58:10.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:10.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:10.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6790/0 conn(0x7fe2e0002840 legacy=0x7fe2e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.317+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6790/0 conn(0x7fe2e0002840 legacy=0x7fe2e0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:10.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe306d1c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:58:10.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe2e0007530 con 0x7fe2e00068e0 2024-09-18T11:58:10.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe304ab8700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6790/0 conn(0x7fe2e00068e0 legacy=0x7fe2e0006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 10 client.104414.objecter ms_handle_connect 0x7fe2e00068e0 2024-09-18T11:58:10.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 10 client.104414.objecter resend_mon_ops 2024-09-18T11:58:10.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955122515 0 0) 0x7fe2e0007530 con 0x7fe2e00068e0 2024-09-18T11:58:10.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 10 monclient(hunting): my global_id is 104420 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e0006590 con 0x7fe2e00068e0 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1281560729 0 0) 0x7fe2e0006590 con 0x7fe2e00068e0 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2e0007530 con 0x7fe2e00068e0 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe2f0005ba0 con 0x7fe2e00068e0 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 10 monclient: dump: 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.338 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2830147565 0 0) 0x7fe2e0007530 con 0x7fe2e00068e0 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.329+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe2e0008760 con 0x7fe2e00068e0 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 2724792113) 0x7fe2f0006fb0 con 0x7fe2e00068e0 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe2fcff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe2fcff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:10.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6790/0 conn(0x7fe2e00068e0 legacy=0x7fe2e0006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe2fcff9700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6790/0 conn(0x7fe2e00068e0 legacy=0x7fe2e0006d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:10.340 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:58:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 60, 2024-09-18T11:58:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:58:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:58:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:58:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:58:10.352 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:10.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 10 librados: watch_flush enter 2024-09-18T11:58:10.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 10 librados: watch_flush exit 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 20 client.104414.objecter shutdown clearing up homeless session... 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 10 client.104414.objecter successfully canceled tick 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.129:6812/20806 conn(0x7fe2ec038320 legacy=0x7fe2ec03a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 10 monclient: shutdown 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 >> v1:172.21.15.112:6790/0 conn(0x7fe300132e40 legacy=0x7fe30014a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe3052b9700 1 -- 172.21.15.112:0/221553651 reap_dead start 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe3052b9700 1 -- 172.21.15.112:0/221553651 reap_dead start 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 shutdown_connections 2024-09-18T11:58:10.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 >> 172.21.15.112:0/221553651 conn(0x7fe3000a5b70 msgr2=0x7fe3000a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:10.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 shutdown_connections 2024-09-18T11:58:10.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 1 -- 172.21.15.112:0/221553651 wait complete. 2024-09-18T11:58:10.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.333+0000 7fe306d1c700 1 librados: shutdown 2024-09-18T11:58:10.356 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:58:10.356 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:58:10.356 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:58:10.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient: get_monmap_and_config 2024-09-18T11:58:10.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient: build_initial_monmap 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient: monmap: 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:10.469553+0000 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:10.469553+0000 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:10.469 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 1 Processor -- start 2024-09-18T11:58:10.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 1 -- start start 2024-09-18T11:58:10.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient: init 2024-09-18T11:58:10.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:10.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:10.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient(hunting): picked mon.noname-a con 0x7f5fec0a0380 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:10.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient(hunting): picked mon.noname-c con 0x7f5fec0a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:10.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient(hunting): picked mon.noname-b con 0x7f5fec097970 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:10.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec097550 con 0x7f5fec0a0380 2024-09-18T11:58:10.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec096cb0 con 0x7f5fec0a0e60 2024-09-18T11:58:10.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0a1440 con 0x7f5fec097970 2024-09-18T11:58:10.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:10.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff3333700 10 monclient(hunting): authenticate will time out at 1662.311287s 2024-09-18T11:58:10.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff2b32700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5fec097970 0x7f5fec13aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47672/0 (socket says 172.21.15.112:47672) 2024-09-18T11:58:10.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff2331700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5fec0a0380 0x7f5fec0a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35026/0 (socket says 172.21.15.112:35026) 2024-09-18T11:58:10.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.465+0000 7f5ff2b32700 1 -- 172.21.15.112:0/3926779223 learned_addr learned my addr 172.21.15.112:0/3926779223 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:10.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff2331700 1 -- 172.21.15.112:0/3926779223 >> v1:172.21.15.112:6789/0 conn(0x7f5fec0a0380 legacy=0x7f5fec0a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff2b32700 1 -- 172.21.15.112:0/3926779223 >> v1:172.21.15.112:6790/0 conn(0x7f5fec097970 legacy=0x7f5fec13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff1b30700 1 -- 172.21.15.112:0/3926779223 >> v1:172.21.15.129:6789/0 conn(0x7f5fec0a0e60 legacy=0x7f5fec137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583278733 0 0) 0x7f5fec097550 con 0x7f5fec0a0380 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient(hunting): my global_id is 134115 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd8003570 con 0x7f5fec0a0380 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20575686 0 0) 0x7f5fec0a1440 con 0x7f5fec097970 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient(hunting): my global_id is 104423 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec097550 con 0x7f5fec097970 2024-09-18T11:58:10.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2512868235 0 0) 0x7f5fec097550 con 0x7f5fec097970 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 >> v1:172.21.15.129:6789/0 conn(0x7f5fec0a0e60 legacy=0x7f5fec137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 >> v1:172.21.15.112:6789/0 conn(0x7f5fec0a0380 legacy=0x7f5fec0a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: _finish_hunting 0 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 monclient: found mon.noname-b 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec0a1de0 con 0x7f5fec097970 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: _finish_auth 0 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fe4001c90 con 0x7f5fec097970 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff3333700 5 monclient: authenticate success, global_id 104423 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff3333700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:10.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: dump: 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff3333700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:10.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fe4001e70 con 0x7f5fec097970 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 1 -- 172.21.15.112:0/3926779223 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fe4004c70 con 0x7f5fec097970 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff132f700 10 monclient: dump: 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.478 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.479 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff3333700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:10.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.469+0000 7f5ff3333700 4 set_mon_vals no callback set 2024-09-18T11:58:10.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient: shutdown 2024-09-18T11:58:10.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- 172.21.15.112:0/3926779223 >> v1:172.21.15.112:6790/0 conn(0x7f5fec097970 legacy=0x7f5fec13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- 172.21.15.112:0/3926779223 shutdown_connections 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- 172.21.15.112:0/3926779223 >> 172.21.15.112:0/3926779223 conn(0x7f5fec068100 msgr2=0x7f5fec0a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- 172.21.15.112:0/3926779223 shutdown_connections 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- 172.21.15.112:0/3926779223 wait complete. 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient: build_initial_monmap 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient: monmap: 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:10.484 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 librados: starting msgr at 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 librados: starting objecter 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 Processor -- start 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- start start 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 librados: setting wanted keys 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 librados: calling monclient init 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient: init 2024-09-18T11:58:10.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient(hunting): picked mon.noname-a con 0x7f5fec097970 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient(hunting): picked mon.noname-c con 0x7f5fec0a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient(hunting): picked mon.noname-b con 0x7f5fec0a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0a2050 con 0x7f5fec097970 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec098560 con 0x7f5fec0a0e60 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec11c880 con 0x7f5fec0a0380 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff3333700 10 monclient(hunting): authenticate will time out at 1662.319104s 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff1b30700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5fec0a0380 0x7f5fec139860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47678/0 (socket says 172.21.15.112:47678) 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff1b30700 1 -- 172.21.15.112:0/4163615727 learned_addr learned my addr 172.21.15.112:0/4163615727 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:10.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.473+0000 7f5ff2331700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5fec097970 0x7f5fec139150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35030/0 (socket says 172.21.15.112:35030) 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff2331700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.112:6789/0 conn(0x7f5fec097970 legacy=0x7f5fec139150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 client.?.objecter ms_handle_connect 0x7f5fec097970 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff1b30700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.112:6790/0 conn(0x7f5fec0a0380 legacy=0x7f5fec139860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 client.?.objecter ms_handle_connect 0x7f5fec0a0380 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 client.?.objecter ms_handle_connect 0x7f5fec0a0e60 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff2b32700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6789/0 conn(0x7f5fec0a0e60 legacy=0x7f5fec149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125092898 0 0) 0x7f5fec0a2050 con 0x7f5fec097970 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): my global_id is 134118 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd0004f10 con 0x7f5fec097970 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794794291 0 0) 0x7f5fec11c880 con 0x7f5fec0a0380 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): my global_id is 104426 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec0a2050 con 0x7f5fec0a0380 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 701632436 0 0) 0x7f5fec0a2050 con 0x7f5fec0a0380 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fec11c880 con 0x7f5fec0a0380 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fdc001ca0 con 0x7f5fec0a0380 2024-09-18T11:58:10.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): dump: 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154076765 0 0) 0x7f5fd0004f10 con 0x7f5fec097970 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fec0a2050 con 0x7f5fec097970 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fe8001bd0 con 0x7f5fec097970 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient(hunting): dump: 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.490 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 523700268 0 0) 0x7f5fec11c880 con 0x7f5fec0a0380 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6789/0 conn(0x7f5fec0a0e60 legacy=0x7f5fec149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.112:6789/0 conn(0x7f5fec097970 legacy=0x7f5fec139150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient: _finish_hunting 0 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 monclient: found mon.c 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec139f70 con 0x7f5fec0a0380 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient: _finish_auth 0 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:10.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 5 monclient: authenticate success, global_id 104426 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 10 monclient: _renew_subs 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec11c5a0 con 0x7f5fec0a0380 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 10 client.104426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 10 monclient: _renew_subs 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec13b7d0 con 0x7f5fec0a0380 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff3333700 1 librados: init done 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fdc0022b0 con 0x7f5fec0a0380 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fdc0025c0 con 0x7f5fec0a0380 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5ff0b2e700 4 set_mon_vals no callback set 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.477+0000 7f5fe27fc700 10 monclient: dump: 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5ff3333700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:10.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5fd4006590 con 0x7f5fd4002840 2024-09-18T11:58:10.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5fdc0027d0 con 0x7f5fec0a0380 2024-09-18T11:58:10.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(534..534 src has 1..534) v4 ==== 5737+0+0 (unknown 1484240301 0 0) 0x7f5fdc044000 con 0x7f5fec0a0380 2024-09-18T11:58:10.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 10 client.104426.objecter ms_dispatch 0x7f5fec000fd0 osd_map(534..534 src has 1..534) v4 2024-09-18T11:58:10.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 3 client.104426.objecter handle_osd_map got epochs [534,534] > 0 2024-09-18T11:58:10.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 3 client.104426.objecter handle_osd_map decoding full epoch 534 2024-09-18T11:58:10.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 20 client.104426.objecter dump_active .. 0 homeless 2024-09-18T11:58:10.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5ff2b32700 10 monclient: get_auth_request con 0x7f5fd0038300 auth_method 0 2024-09-18T11:58:10.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 10 client.104426.objecter ms_handle_connect 0x7f5fd4002840 2024-09-18T11:58:10.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 10 client.104426.objecter resend_mon_ops 2024-09-18T11:58:10.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5ff2331700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6789/0 conn(0x7f5fd4002840 legacy=0x7f5fd4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.481+0000 7f5fe27fc700 10 client.104426.objecter ms_handle_connect 0x7f5fd0038300 2024-09-18T11:58:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258991288 0 0) 0x7f5fd4006590 con 0x7f5fd4002840 2024-09-18T11:58:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 10 monclient(hunting): my global_id is 114313 2024-09-18T11:58:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec11c880 con 0x7f5fd4002840 2024-09-18T11:58:10.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3816077533 0 0) 0x7f5fec11c880 con 0x7f5fd4002840 2024-09-18T11:58:10.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fd4006590 con 0x7f5fd4002840 2024-09-18T11:58:10.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fe8004eb0 con 0x7f5fd4002840 2024-09-18T11:58:10.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 10 monclient: dump: 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3547331369 0 0) 0x7f5fd4006590 con 0x7f5fd4002840 2024-09-18T11:58:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fd400a140 con 0x7f5fd4002840 2024-09-18T11:58:10.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5fe80050f0 con 0x7f5fd4002840 2024-09-18T11:58:10.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:10.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:10.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6789/0 conn(0x7f5fd4002840 legacy=0x7f5fd4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.909+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6789/0 conn(0x7f5fd4002840 legacy=0x7f5fd4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:10.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5ff3333700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:58:10.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5fd4007550 con 0x7f5fd4006900 2024-09-18T11:58:10.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5fe27fc700 10 client.104426.objecter ms_handle_connect 0x7f5fd4006900 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5fe27fc700 10 client.104426.objecter resend_mon_ops 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5ff2331700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6789/0 conn(0x7f5fd4006900 legacy=0x7f5fd4006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020956328 0 0) 0x7f5fd4007550 con 0x7f5fd4006900 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5fe27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5fe27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5fe27fc700 10 monclient(hunting): my global_id is 114316 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.921+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd4006590 con 0x7f5fd4006900 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3540376947 0 0) 0x7f5fd4006590 con 0x7f5fd4006900 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fd4007550 con 0x7f5fd4006900 2024-09-18T11:58:10.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fe8005860 con 0x7f5fd4006900 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 10 monclient: dump: 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:10.930 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:10.930 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:10.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 246739478 0 0) 0x7f5fd4007550 con 0x7f5fd4006900 2024-09-18T11:58:10.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5fd40087b0 con 0x7f5fd4006900 2024-09-18T11:58:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 1337738144) 0x7f5fe8007ac0 con 0x7f5fd4006900 2024-09-18T11:58:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:58:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6789/0 conn(0x7f5fd4006900 legacy=0x7f5fd4006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5fe27fc700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6789/0 conn(0x7f5fd4006900 legacy=0x7f5fd4006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:10.931 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:10.931 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:10.931 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 60, 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:58:10.932 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.933 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:58:10.934 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:10.935 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.936 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.937 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:10.938 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:10.939 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.939 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.939 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.939 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:58:10.939 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.940 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.940 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:58:10.940 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:58:10.940 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.940 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.940 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:58:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:58:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.943 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:10.944 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:10.945 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:10.945 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:10.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:10.945 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:10.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:10.945 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:58:10.945 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 10 librados: watch_flush enter 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 10 librados: watch_flush exit 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 20 client.104426.objecter shutdown clearing up homeless session... 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 10 client.104426.objecter successfully canceled tick 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.129:6812/20806 conn(0x7f5fd0038300 legacy=0x7f5fd003a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 10 monclient: shutdown 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 >> v1:172.21.15.112:6790/0 conn(0x7f5fec0a0380 legacy=0x7f5fec139860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:10.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff2b32700 1 -- 172.21.15.112:0/4163615727 reap_dead start 2024-09-18T11:58:10.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff2b32700 1 -- 172.21.15.112:0/4163615727 reap_dead start 2024-09-18T11:58:10.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 shutdown_connections 2024-09-18T11:58:10.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 >> 172.21.15.112:0/4163615727 conn(0x7f5fec068100 msgr2=0x7f5fec0a4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:10.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 shutdown_connections 2024-09-18T11:58:10.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 1 -- 172.21.15.112:0/4163615727 wait complete. 2024-09-18T11:58:10.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:10.925+0000 7f5ff3333700 1 librados: shutdown 2024-09-18T11:58:10.948 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:58:10.948 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T11:58:10.948 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.049+0000 7fda6ec90700 10 monclient: get_monmap_and_config 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.049+0000 7fda6ec90700 10 monclient: build_initial_monmap 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.049+0000 7fda6ec90700 10 monclient: monmap: 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:11.055978+0000 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:11.055978+0000 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:11.056 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:11.056 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:11.056 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 1 Processor -- start 2024-09-18T11:58:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 1 -- start start 2024-09-18T11:58:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 10 monclient: init 2024-09-18T11:58:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:58:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 10 monclient(hunting): picked mon.noname-c con 0x7fda680a2270 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 10 monclient(hunting): picked mon.noname-a con 0x7fda6806aa70 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 10 monclient(hunting): picked mon.noname-b con 0x7fda6806cb80 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6806a830 con 0x7fda6806aa70 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda680a2640 con 0x7fda680a2270 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6806d000 con 0x7fda6806cb80 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ec90700 10 monclient(hunting): authenticate will time out at 1662.896890s 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6d22d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fda6806cb80 0x7fda68137100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47706/0 (socket says 172.21.15.112:47706) 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6d22d700 1 -- 172.21.15.112:0/2571190592 learned_addr learned my addr 172.21.15.112:0/2571190592 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5ffff700 1 --1- 172.21.15.112:0/2571190592 >> v1:172.21.15.112:6789/0 conn(0x7fda6806aa70 0x7fda680a0780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35042/0 (socket says 172.21.15.112:35042) 2024-09-18T11:58:11.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6d22d700 1 -- 172.21.15.112:0/2571190592 >> v1:172.21.15.112:6790/0 conn(0x7fda6806cb80 legacy=0x7fda68137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:11.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5ffff700 1 -- 172.21.15.112:0/2571190592 >> v1:172.21.15.112:6789/0 conn(0x7fda6806aa70 legacy=0x7fda680a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:11.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda6ca2c700 1 -- 172.21.15.112:0/2571190592 >> v1:172.21.15.129:6789/0 conn(0x7fda680a2270 legacy=0x7fda680a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:11.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452576711 0 0) 0x7fda6806d000 con 0x7fda6806cb80 2024-09-18T11:58:11.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:11.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:11.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): my global_id is 104429 2024-09-18T11:58:11.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda50003570 con 0x7fda6806cb80 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709366403 0 0) 0x7fda6806a830 con 0x7fda6806aa70 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): my global_id is 134121 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6806d000 con 0x7fda6806aa70 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506932155 0 0) 0x7fda680a2640 con 0x7fda680a2270 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.053+0000 7fda5f7fe700 10 monclient(hunting): my global_id is 114319 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6806a830 con 0x7fda680a2270 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2785330167 0 0) 0x7fda50003570 con 0x7fda6806cb80 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 >> v1:172.21.15.129:6789/0 conn(0x7fda680a2270 legacy=0x7fda680a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 >> v1:172.21.15.112:6789/0 conn(0x7fda6806aa70 legacy=0x7fda680a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:11.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 monclient: found mon.noname-b 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda680a0e90 con 0x7fda6806cb80 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: _finish_auth 0 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fda60001cc0 con 0x7fda6806cb80 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 5 monclient: authenticate success, global_id 104429 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: dump: 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:11.063 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda60001f90 con 0x7fda6806cb80 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 1 -- 172.21.15.112:0/2571190592 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fda60004e10 con 0x7fda6806cb80 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:11.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda5f7fe700 10 monclient: dump: 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:11.065 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:11.066 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:11.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:11.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 4 set_mon_vals no callback set 2024-09-18T11:58:11.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 10 monclient: shutdown 2024-09-18T11:58:11.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:11.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 1 -- 172.21.15.112:0/2571190592 >> v1:172.21.15.112:6790/0 conn(0x7fda6806cb80 legacy=0x7fda68137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:11.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 1 -- 172.21.15.112:0/2571190592 shutdown_connections 2024-09-18T11:58:11.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 1 -- 172.21.15.112:0/2571190592 >> 172.21.15.112:0/2571190592 conn(0x7fda680680d0 msgr2=0x7fda680a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:11.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 1 -- 172.21.15.112:0/2571190592 shutdown_connections 2024-09-18T11:58:11.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 1 -- 172.21.15.112:0/2571190592 wait complete. 2024-09-18T11:58:11.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 10 monclient: build_initial_monmap 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.057+0000 7fda6ec90700 10 monclient: monmap: 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 librados: starting msgr at 2024-09-18T11:58:11.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 librados: starting objecter 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 Processor -- start 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 -- start start 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 librados: setting wanted keys 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 librados: calling monclient init 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 10 monclient: init 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 10 monclient(hunting): picked mon.noname-c con 0x7fda6806aa70 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 10 monclient(hunting): picked mon.noname-a con 0x7fda6806cb80 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 10 monclient(hunting): picked mon.noname-b con 0x7fda680a2270 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6809d7b0 con 0x7fda6806cb80 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6811b870 con 0x7fda680a2270 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda68119a60 con 0x7fda6806aa70 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ec90700 10 monclient(hunting): authenticate will time out at 1662.905740s 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ca2c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fda6806aa70 0x7fda6813fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47720/0 (socket says 172.21.15.112:47720) 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.061+0000 7fda6ca2c700 1 -- 172.21.15.112:0/2918255031 learned_addr learned my addr 172.21.15.112:0/2918255031 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ca2c700 1 -- 172.21.15.112:0/2918255031 >> v1:172.21.15.112:6790/0 conn(0x7fda6806aa70 legacy=0x7fda6813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 client.?.objecter ms_handle_connect 0x7fda6806aa70 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 client.?.objecter ms_handle_connect 0x7fda6806cb80 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5ffff700 1 -- 172.21.15.112:0/2918255031 >> v1:172.21.15.112:6789/0 conn(0x7fda6806cb80 legacy=0x7fda68144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154425785 0 0) 0x7fda68119a60 con 0x7fda6806aa70 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): my global_id is 104432 2024-09-18T11:58:11.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6d22d700 1 -- 172.21.15.112:0/2918255031 >> v1:172.21.15.129:6789/0 conn(0x7fda680a2270 legacy=0x7fda68147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda54004f10 con 0x7fda6806aa70 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 client.?.objecter ms_handle_connect 0x7fda680a2270 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384810661 0 0) 0x7fda6809d7b0 con 0x7fda6806cb80 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): my global_id is 134124 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda68119a60 con 0x7fda6806cb80 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764885928 0 0) 0x7fda6811b870 con 0x7fda680a2270 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): my global_id is 114322 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6809d7b0 con 0x7fda680a2270 2024-09-18T11:58:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 328737322 0 0) 0x7fda54004f10 con 0x7fda6806aa70 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda6811b870 con 0x7fda6806aa70 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fda64001cd0 con 0x7fda6806aa70 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): dump: 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3464562280 0 0) 0x7fda68119a60 con 0x7fda6806cb80 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda54004f10 con 0x7fda6806cb80 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fda58004830 con 0x7fda6806cb80 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient(hunting): dump: 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.077 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2923916038 0 0) 0x7fda6811b870 con 0x7fda6806aa70 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 >> v1:172.21.15.129:6789/0 conn(0x7fda680a2270 legacy=0x7fda68147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 >> v1:172.21.15.112:6789/0 conn(0x7fda6806cb80 legacy=0x7fda68144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient: _finish_hunting 0 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 monclient: found mon.c 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6811b590 con 0x7fda6806aa70 2024-09-18T11:58:11.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient: _finish_auth 0 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 5 monclient: authenticate success, global_id 104432 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 10 monclient: _renew_subs 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda68140340 con 0x7fda6806aa70 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 10 client.104432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 10 monclient: _renew_subs 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda681317d0 con 0x7fda6806aa70 2024-09-18T11:58:11.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 1 librados: init done 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda64002360 con 0x7fda6806aa70 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fda64002730 con 0x7fda6806aa70 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5effd700 4 set_mon_vals no callback set 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 monclient: dump: 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:11.080 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fda64002890 con 0x7fda6806aa70 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda480050b0 con 0x7fda6806aa70 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(535..535 src has 1..535) v4 ==== 5364+0+0 (unknown 862203077 0 0) 0x7fda64043f50 con 0x7fda6806aa70 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 10 client.104432.objecter ms_dispatch 0x7fda68000fd0 osd_map(535..535 src has 1..535) v4 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 3 client.104432.objecter handle_osd_map got epochs [535,535] > 0 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 3 client.104432.objecter handle_osd_map decoding full epoch 535 2024-09-18T11:58:11.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5cff9700 20 client.104432.objecter dump_active .. 0 homeless 2024-09-18T11:58:11.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.065+0000 7fda5ffff700 10 monclient: get_auth_request con 0x7fda54038280 auth_method 0 2024-09-18T11:58:11.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.069+0000 7fda5cff9700 10 client.104432.objecter ms_handle_connect 0x7fda54038280 2024-09-18T11:58:11.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.073+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fda64002b40 con 0x7fda6806aa70 2024-09-18T11:58:11.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.073+0000 7fda5cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:11.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.073+0000 7fda5cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:11.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.297+0000 7fda6ec90700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T11:58:11.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.297+0000 7fda6ec90700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T11:58:11.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.297+0000 7fda6ec90700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:11.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.297+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fda48005c30 con 0x7fda6806aa70 2024-09-18T11:58:11.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fda5cff9700 1 -- 172.21.15.112:0/2918255031 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fda640145c0 con 0x7fda6806aa70 2024-09-18T11:58:11.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fda5cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T11:58:11.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fda5cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:11.307 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.307 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=8,logm=58} crc {auth=1751392425,config=3140514009,health=4153959047,kv=1310296871,logm=33837928}) 2024-09-18T11:58:11.307 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=8,logm=60} crc {auth=4032096812,config=3140514009,health=2373925959,kv=1310296871,logm=1972369495}) 2024-09-18T11:58:11.307 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.307 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=8,logm=58} crc {auth=1751392425,config=3140514009,health=4153959047,kv=1310296871,logm=33837928}) 2024-09-18T11:58:11.307 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.301+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=8,logm=60} crc {auth=2282583964,config=3140514009,health=1903608717,kv=1310296871,logm=1156167215}) 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 10 librados: watch_flush enter 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 10 librados: watch_flush exit 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 20 client.104432.objecter shutdown clearing up homeless session... 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 10 client.104432.objecter successfully canceled tick 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 >> v1:172.21.15.129:6812/20806 conn(0x7fda54038280 legacy=0x7fda5403a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 10 monclient: shutdown 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 >> v1:172.21.15.112:6790/0 conn(0x7fda6806aa70 legacy=0x7fda6813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 shutdown_connections 2024-09-18T11:58:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 >> 172.21.15.112:0/2918255031 conn(0x7fda680680d0 msgr2=0x7fda6809e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:11.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 shutdown_connections 2024-09-18T11:58:11.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 1 -- 172.21.15.112:0/2918255031 wait complete. 2024-09-18T11:58:11.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fda6ec90700 1 librados: shutdown 2024-09-18T11:58:11.311 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.311 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663411543}) 2024-09-18T11:58:11.311 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4219332918}) 2024-09-18T11:58:11.311 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.311 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663411543}) 2024-09-18T11:58:11.311 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.305+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334057340}) 2024-09-18T11:58:11.313 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.309+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.313 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.309+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536108136}) 2024-09-18T11:58:11.313 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.309+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4203450861}) 2024-09-18T11:58:11.313 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.309+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.314 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.309+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536108136}) 2024-09-18T11:58:11.314 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.309+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894204522}) 2024-09-18T11:58:11.316 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.316 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139159164}) 2024-09-18T11:58:11.316 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=88773}) 2024-09-18T11:58:11.316 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.316 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139159164}) 2024-09-18T11:58:11.316 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3894069447}) 2024-09-18T11:58:11.317 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.317 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014243928}) 2024-09-18T11:58:11.318 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3438173661}) 2024-09-18T11:58:11.318 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.318 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014243928}) 2024-09-18T11:58:11.318 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445242814}) 2024-09-18T11:58:11.319 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.319 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3721491682}) 2024-09-18T11:58:11.319 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2616907186}) 2024-09-18T11:58:11.319 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.319 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3721491682}) 2024-09-18T11:58:11.319 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.313+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507489368}) 2024-09-18T11:58:11.320 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.320 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1540809151}) 2024-09-18T11:58:11.320 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016707887}) 2024-09-18T11:58:11.320 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.321 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1540809151}) 2024-09-18T11:58:11.321 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=212339145}) 2024-09-18T11:58:11.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T11:58:11.322 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.322 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053101280}) 2024-09-18T11:58:11.323 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1462658651}) 2024-09-18T11:58:11.323 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.323 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053101280}) 2024-09-18T11:58:11.323 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.317+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4261811214}) 2024-09-18T11:58:11.324 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.324 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582057149}) 2024-09-18T11:58:11.324 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198528785}) 2024-09-18T11:58:11.324 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.324 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582057149}) 2024-09-18T11:58:11.324 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2203063541}) 2024-09-18T11:58:11.325 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.325 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094055755}) 2024-09-18T11:58:11.325 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091195389}) 2024-09-18T11:58:11.326 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.326 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094055755}) 2024-09-18T11:58:11.326 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=606385514}) 2024-09-18T11:58:11.327 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.327 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593627523}) 2024-09-18T11:58:11.327 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228754040}) 2024-09-18T11:58:11.327 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.327 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593627523}) 2024-09-18T11:58:11.327 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.321+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918967878}) 2024-09-18T11:58:11.328 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.328 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361854783}) 2024-09-18T11:58:11.328 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2595036661}) 2024-09-18T11:58:11.328 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.328 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361854783}) 2024-09-18T11:58:11.328 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047309710}) 2024-09-18T11:58:11.330 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.330 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556164164}) 2024-09-18T11:58:11.330 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750922917}) 2024-09-18T11:58:11.330 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.330 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556164164}) 2024-09-18T11:58:11.330 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2833101563}) 2024-09-18T11:58:11.330 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.330 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964250686}) 2024-09-18T11:58:11.331 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830577490}) 2024-09-18T11:58:11.331 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.331 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964250686}) 2024-09-18T11:58:11.331 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3999012075}) 2024-09-18T11:58:11.332 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.332 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401496609}) 2024-09-18T11:58:11.332 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968068105}) 2024-09-18T11:58:11.332 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.332 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.325+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401496609}) 2024-09-18T11:58:11.332 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399407208}) 2024-09-18T11:58:11.333 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.333 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211147818}) 2024-09-18T11:58:11.333 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1074810581}) 2024-09-18T11:58:11.333 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.333 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211147818}) 2024-09-18T11:58:11.333 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=311501329}) 2024-09-18T11:58:11.334 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.334 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014750404}) 2024-09-18T11:58:11.334 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1363395163}) 2024-09-18T11:58:11.334 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.334 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014750404}) 2024-09-18T11:58:11.334 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1704407503}) 2024-09-18T11:58:11.335 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.335 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=112396426}) 2024-09-18T11:58:11.335 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1526399215}) 2024-09-18T11:58:11.335 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.335 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=112396426}) 2024-09-18T11:58:11.335 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.329+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=992514703}) 2024-09-18T11:58:11.336 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.336 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2327604350}) 2024-09-18T11:58:11.336 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2852701412}) 2024-09-18T11:58:11.337 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.337 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2327604350}) 2024-09-18T11:58:11.337 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3097998689}) 2024-09-18T11:58:11.338 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.338 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357023731}) 2024-09-18T11:58:11.338 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3628578856}) 2024-09-18T11:58:11.338 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.338 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357023731}) 2024-09-18T11:58:11.338 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931330857}) 2024-09-18T11:58:11.339 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.339 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570120923}) 2024-09-18T11:58:11.339 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=400660189}) 2024-09-18T11:58:11.339 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.339 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570120923}) 2024-09-18T11:58:11.339 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.333+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3200299770}) 2024-09-18T11:58:11.340 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.340 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2403395633}) 2024-09-18T11:58:11.340 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=192631804}) 2024-09-18T11:58:11.341 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.341 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2403395633}) 2024-09-18T11:58:11.341 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1012326890}) 2024-09-18T11:58:11.342 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.342 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218393574}) 2024-09-18T11:58:11.342 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532039607}) 2024-09-18T11:58:11.342 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.342 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218393574}) 2024-09-18T11:58:11.343 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3700052302}) 2024-09-18T11:58:11.343 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.343 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996726783}) 2024-09-18T11:58:11.343 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272086984}) 2024-09-18T11:58:11.343 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.343 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996726783}) 2024-09-18T11:58:11.343 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.337+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047801673}) 2024-09-18T11:58:11.344 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.344 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393535198}) 2024-09-18T11:58:11.344 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511310527}) 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393535198}) 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898331423}) 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2164102684}) 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=866781532}) 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2164102684}) 2024-09-18T11:58:11.345 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4203458474}) 2024-09-18T11:58:11.346 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045557966}) 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3370359282}) 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045557966}) 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1341106119}) 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555494979}) 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837894457}) 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555494979}) 2024-09-18T11:58:11.347 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.341+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842198219}) 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=681873446}) 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1594619899}) 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=681873446}) 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2297186849}) 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200230923}) 2024-09-18T11:58:11.349 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=999433230}) 2024-09-18T11:58:11.350 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.350 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200230923}) 2024-09-18T11:58:11.350 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515739291}) 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783224899}) 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2219078549}) 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783224899}) 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2352321132}) 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932224702}) 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372011521}) 2024-09-18T11:58:11.351 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.352 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932224702}) 2024-09-18T11:58:11.352 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.345+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180436813}) 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944048372}) 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1455445380}) 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944048372}) 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595686817}) 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194089645}) 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607467950}) 2024-09-18T11:58:11.353 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.354 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194089645}) 2024-09-18T11:58:11.354 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2914994338}) 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934678727}) 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3643758370}) 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934678727}) 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=757051344}) 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481365281}) 2024-09-18T11:58:11.355 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275205427}) 2024-09-18T11:58:11.356 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.356 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481365281}) 2024-09-18T11:58:11.356 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.349+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=927759494}) 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218254121}) 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1887461466}) 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218254121}) 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1293806703}) 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840885481}) 2024-09-18T11:58:11.357 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968627687}) 2024-09-18T11:58:11.358 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.358 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840885481}) 2024-09-18T11:58:11.358 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2147654121}) 2024-09-18T11:58:11.359 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.359 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217357370}) 2024-09-18T11:58:11.359 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=811159781}) 2024-09-18T11:58:11.359 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.359 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217357370}) 2024-09-18T11:58:11.360 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069944954}) 2024-09-18T11:58:11.360 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.360 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337605400}) 2024-09-18T11:58:11.360 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686497107}) 2024-09-18T11:58:11.360 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.360 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337605400}) 2024-09-18T11:58:11.360 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.353+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941452949}) 2024-09-18T11:58:11.361 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.361 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1820547553}) 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187179864}) 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1820547553}) 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234640574}) 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714635735}) 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272152719}) 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714635735}) 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831966885}) 2024-09-18T11:58:11.362 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.363 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242705604}) 2024-09-18T11:58:11.363 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236846365}) 2024-09-18T11:58:11.363 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.363 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242705604}) 2024-09-18T11:58:11.363 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278636396}) 2024-09-18T11:58:11.364 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.364 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496833348}) 2024-09-18T11:58:11.364 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3626278104}) 2024-09-18T11:58:11.364 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.364 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496833348}) 2024-09-18T11:58:11.364 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.357+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2517477757}) 2024-09-18T11:58:11.373 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.373 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972456471}) 2024-09-18T11:58:11.373 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832039952}) 2024-09-18T11:58:11.373 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.373 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972456471}) 2024-09-18T11:58:11.373 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652923967}) 2024-09-18T11:58:11.374 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.374 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481061136}) 2024-09-18T11:58:11.374 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2682562430}) 2024-09-18T11:58:11.374 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.375 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481061136}) 2024-09-18T11:58:11.375 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.369+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1825930271}) 2024-09-18T11:58:11.385 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.381+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.385 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.381+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1832349500}) 2024-09-18T11:58:11.385 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.381+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327824488}) 2024-09-18T11:58:11.385 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.381+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.385 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.381+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1832349500}) 2024-09-18T11:58:11.386 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.381+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290630080}) 2024-09-18T11:58:11.387 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.385+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.387 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.385+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388958115}) 2024-09-18T11:58:11.387 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.385+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2079011242}) 2024-09-18T11:58:11.388 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.385+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.388 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.385+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388958115}) 2024-09-18T11:58:11.388 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.385+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79233772}) 2024-09-18T11:58:11.393 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.389+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.393 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.389+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771150055}) 2024-09-18T11:58:11.394 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.389+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1276449063}) 2024-09-18T11:58:11.394 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.389+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.394 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.389+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771150055}) 2024-09-18T11:58:11.394 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.389+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3784841364}) 2024-09-18T11:58:11.907 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.901+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.907 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.901+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964866998}) 2024-09-18T11:58:11.907 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.901+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11634910}) 2024-09-18T11:58:11.907 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.901+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.907 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.901+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964866998}) 2024-09-18T11:58:11.907 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.901+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=228273180}) 2024-09-18T11:58:11.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.921+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.921+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2976462329}) 2024-09-18T11:58:11.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.921+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2666828605}) 2024-09-18T11:58:11.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.921+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.921+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2976462329}) 2024-09-18T11:58:11.925 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.921+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3676134581}) 2024-09-18T11:58:11.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.929+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.929+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009888937}) 2024-09-18T11:58:11.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.929+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2289920726}) 2024-09-18T11:58:11.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.929+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.929+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009888937}) 2024-09-18T11:58:11.932 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.929+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403403420}) 2024-09-18T11:58:11.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.929+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785787311}) 2024-09-18T11:58:11.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582048960}) 2024-09-18T11:58:11.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.935 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785787311}) 2024-09-18T11:58:11.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407765088}) 2024-09-18T11:58:11.937 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.937 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021826209}) 2024-09-18T11:58:11.937 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781852139}) 2024-09-18T11:58:11.937 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021826209}) 2024-09-18T11:58:11.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.933+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3296765837}) 2024-09-18T11:58:11.941 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.937+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.941 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.937+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259598081}) 2024-09-18T11:58:11.941 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.937+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352498896}) 2024-09-18T11:58:11.941 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.937+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:11.941 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.937+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259598081}) 2024-09-18T11:58:11.941 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:11.937+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871990752}) 2024-09-18T11:58:12.072 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.069+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.073 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.069+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3133107557,mdsmap=1531151339,mgr=2470090982}) 2024-09-18T11:58:12.073 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.069+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=3375637741,mdsmap=1531151339,mgr=2436724755}) 2024-09-18T11:58:12.073 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.069+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.073 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.069+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3133107557,mdsmap=1531151339,mgr=2470090982}) 2024-09-18T11:58:12.073 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.069+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1694862398,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=545884692}) 2024-09-18T11:58:12.075 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.075 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=33} crc {mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=913397083,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=1379838495,osd_snap=328679201,osdmap=2789497449}) 2024-09-18T11:58:12.076 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=33} crc {mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=913397083,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=1379838495,osd_snap=2393500091,osdmap=2431695177}) 2024-09-18T11:58:12.076 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.076 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=33} crc {mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=913397083,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=1379838495,osd_snap=328679201,osdmap=2789497449}) 2024-09-18T11:58:12.076 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=36} crc {mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=2106363632,mon_config_key=868353578,monmap=2808122079,osd_metadata=871764258,osd_pg_creating=1379838495,osd_snap=1963539001,osdmap=2564179269}) 2024-09-18T11:58:12.079 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.079 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1123480369}) 2024-09-18T11:58:12.079 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=945074278}) 2024-09-18T11:58:12.079 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.079 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1123480369}) 2024-09-18T11:58:12.079 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.073+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2859895376}) 2024-09-18T11:58:12.081 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.077+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.081 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.077+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1824091127}) 2024-09-18T11:58:12.082 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.077+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2389613408}) 2024-09-18T11:58:12.082 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.077+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.082 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.077+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1824091127}) 2024-09-18T11:58:12.082 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.077+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2939949638}) 2024-09-18T11:58:12.084 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.084 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3840713122}) 2024-09-18T11:58:12.084 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=426127729}) 2024-09-18T11:58:12.084 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.084 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3840713122}) 2024-09-18T11:58:12.084 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2108990000}) 2024-09-18T11:58:12.086 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.086 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=746819408}) 2024-09-18T11:58:12.086 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3197117672}) 2024-09-18T11:58:12.086 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.086 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=746819408}) 2024-09-18T11:58:12.086 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.081+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4002520233}) 2024-09-18T11:58:12.088 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.088 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3952024717}) 2024-09-18T11:58:12.088 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1444922530}) 2024-09-18T11:58:12.088 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.088 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3952024717}) 2024-09-18T11:58:12.088 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3668911866}) 2024-09-18T11:58:12.090 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.090 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=797615339}) 2024-09-18T11:58:12.090 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=674116515}) 2024-09-18T11:58:12.090 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.091 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=797615339}) 2024-09-18T11:58:12.091 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.085+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2407426326}) 2024-09-18T11:58:12.092 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.089+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.089+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=356532344}) 2024-09-18T11:58:12.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.089+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=734455234}) 2024-09-18T11:58:12.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.089+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.089+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=356532344}) 2024-09-18T11:58:12.093 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.089+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1101080161}) 2024-09-18T11:58:12.099 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.097+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.099 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.097+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=1191932395}) 2024-09-18T11:58:12.099 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.097+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=4217806365}) 2024-09-18T11:58:12.100 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.097+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T11:58:12.100 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.097+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=1191932395}) 2024-09-18T11:58:12.100 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T11:58:12.097+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=1409987130}) 2024-09-18T11:58:12.323 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:58:12.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:58:12.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.433+0000 7eff99f34700 10 monclient: get_monmap_and_config 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.433+0000 7eff99f34700 10 monclient: build_initial_monmap 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.433+0000 7eff99f34700 10 monclient: monmap: 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:12.439602+0000 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:12.439602+0000 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:12.439 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 1 Processor -- start 2024-09-18T11:58:12.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 1 -- start start 2024-09-18T11:58:12.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 10 monclient: init 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 10 monclient(hunting): picked mon.noname-c con 0x7eff940a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 10 monclient(hunting): picked mon.noname-a con 0x7eff940a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 10 monclient(hunting): picked mon.noname-b con 0x7eff941385b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff94097270 con 0x7eff940a0750 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff940a3170 con 0x7eff940a0380 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff94096cb0 con 0x7eff941385b0 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99f34700 10 monclient(hunting): authenticate will time out at 1664.281018s 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff93fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7eff940a0750 0x7eff940a0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35070/0 (socket says 172.21.15.112:35070) 2024-09-18T11:58:12.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff93fff700 1 -- 172.21.15.112:0/774525715 learned_addr learned my addr 172.21.15.112:0/774525715 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:12.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff93fff700 1 -- 172.21.15.112:0/774525715 >> v1:172.21.15.112:6789/0 conn(0x7eff940a0750 legacy=0x7eff940a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff99733700 1 -- 172.21.15.112:0/774525715 >> v1:172.21.15.112:6790/0 conn(0x7eff941385b0 legacy=0x7eff9413a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff98f32700 1 -- 172.21.15.112:0/774525715 >> v1:172.21.15.129:6789/0 conn(0x7eff940a0380 legacy=0x7eff94097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252067006 0 0) 0x7eff94097270 con 0x7eff940a0750 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 10 monclient(hunting): my global_id is 134127 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff7c003570 con 0x7eff940a0750 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086378738 0 0) 0x7eff94096cb0 con 0x7eff941385b0 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 10 monclient(hunting): my global_id is 104435 2024-09-18T11:58:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.437+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff94097270 con 0x7eff941385b0 2024-09-18T11:58:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960911012 0 0) 0x7eff940a3170 con 0x7eff940a0380 2024-09-18T11:58:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient(hunting): my global_id is 114325 2024-09-18T11:58:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff94096cb0 con 0x7eff940a0380 2024-09-18T11:58:12.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3281896260 0 0) 0x7eff7c003570 con 0x7eff940a0750 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 >> v1:172.21.15.112:6790/0 conn(0x7eff941385b0 legacy=0x7eff9413a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 >> v1:172.21.15.129:6789/0 conn(0x7eff940a0380 legacy=0x7eff94097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: _finish_hunting 0 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 monclient: found mon.noname-a 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff940a1420 con 0x7eff940a0750 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: _finish_auth 0 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7eff84001d80 con 0x7eff940a0750 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:12.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: dump: 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 5 monclient: authenticate success, global_id 134127 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:12.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff84001f60 con 0x7eff940a0750 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 1 -- 172.21.15.112:0/774525715 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7eff84004e10 con 0x7eff940a0750 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 4 set_mon_vals no callback set 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff937fe700 10 monclient: dump: 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.450 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.452 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.452 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient: shutdown 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- 172.21.15.112:0/774525715 >> v1:172.21.15.112:6789/0 conn(0x7eff940a0750 legacy=0x7eff940a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- 172.21.15.112:0/774525715 shutdown_connections 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- 172.21.15.112:0/774525715 >> 172.21.15.112:0/774525715 conn(0x7eff940a5990 msgr2=0x7eff940a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- 172.21.15.112:0/774525715 shutdown_connections 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- 172.21.15.112:0/774525715 wait complete. 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient: build_initial_monmap 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient: monmap: 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:12.453 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 librados: starting msgr at 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 librados: starting objecter 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 Processor -- start 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- start start 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 librados: setting wanted keys 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 librados: calling monclient init 2024-09-18T11:58:12.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient: init 2024-09-18T11:58:12.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient(hunting): picked mon.noname-b con 0x7eff940a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient(hunting): picked mon.noname-a con 0x7eff940a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient(hunting): picked mon.noname-c con 0x7eff941385b0 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff94138980 con 0x7eff940a0750 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff94098240 con 0x7eff940a0380 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff94098520 con 0x7eff941385b0 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99f34700 10 monclient(hunting): authenticate will time out at 1664.287450s 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99733700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7eff941385b0 0x7eff94149a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47758/0 (socket says 172.21.15.112:47758) 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.441+0000 7eff99733700 1 -- 172.21.15.112:0/3978082397 learned_addr learned my addr 172.21.15.112:0/3978082397 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:12.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99733700 1 -- 172.21.15.112:0/3978082397 >> v1:172.21.15.112:6790/0 conn(0x7eff941385b0 legacy=0x7eff94149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 client.?.objecter ms_handle_connect 0x7eff941385b0 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 client.?.objecter ms_handle_connect 0x7eff940a0750 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff93fff700 1 -- 172.21.15.112:0/3978082397 >> v1:172.21.15.112:6789/0 conn(0x7eff940a0750 legacy=0x7eff9409e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 client.?.objecter ms_handle_connect 0x7eff940a0380 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff98f32700 1 -- 172.21.15.112:0/3978082397 >> v1:172.21.15.129:6789/0 conn(0x7eff940a0380 legacy=0x7eff94141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930558938 0 0) 0x7eff94098520 con 0x7eff941385b0 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): my global_id is 104438 2024-09-18T11:58:12.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff80004f10 con 0x7eff941385b0 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583098974 0 0) 0x7eff94138980 con 0x7eff940a0750 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): my global_id is 134130 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff94098520 con 0x7eff940a0750 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076018967 0 0) 0x7eff94098240 con 0x7eff940a0380 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): my global_id is 114328 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff94138980 con 0x7eff940a0380 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3768413218 0 0) 0x7eff80004f10 con 0x7eff941385b0 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff94098240 con 0x7eff941385b0 2024-09-18T11:58:12.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7eff8c001cd0 con 0x7eff941385b0 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): dump: 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034446252 0 0) 0x7eff94098520 con 0x7eff940a0750 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff80004f10 con 0x7eff940a0750 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7eff840016a0 con 0x7eff940a0750 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): dump: 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3773919997 0 0) 0x7eff94138980 con 0x7eff940a0380 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff94098520 con 0x7eff940a0380 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7eff88001c40 con 0x7eff940a0380 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient(hunting): dump: 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.465 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4194820577 0 0) 0x7eff94098240 con 0x7eff941385b0 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 >> v1:172.21.15.129:6789/0 conn(0x7eff940a0380 legacy=0x7eff94141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 >> v1:172.21.15.112:6789/0 conn(0x7eff940a0750 legacy=0x7eff9409e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:58:12.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 monclient: found mon.c 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff94141d70 con 0x7eff941385b0 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient: _finish_auth 0 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 5 monclient: authenticate success, global_id 104438 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 10 monclient: _renew_subs 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9409e760 con 0x7eff941385b0 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 10 client.104438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 10 monclient: _renew_subs 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff94137f90 con 0x7eff941385b0 2024-09-18T11:58:12.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff99f34700 1 librados: init done 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff8c0021b0 con 0x7eff941385b0 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7eff8c002500 con 0x7eff941385b0 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff927fc700 4 set_mon_vals no callback set 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.445+0000 7eff90ff9700 10 monclient: dump: 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.468 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff99f34700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff99f34700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff99f34700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff740050b0 con 0x7eff941385b0 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7eff8c002660 con 0x7eff941385b0 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(536..536 src has 1..536) v4 ==== 5737+0+0 (unknown 449751968 0 0) 0x7eff8c043fa0 con 0x7eff941385b0 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff90ff9700 10 client.104438.objecter ms_dispatch 0x7eff94000fd0 osd_map(536..536 src has 1..536) v4 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff90ff9700 3 client.104438.objecter handle_osd_map got epochs [536,536] > 0 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff90ff9700 3 client.104438.objecter handle_osd_map decoding full epoch 536 2024-09-18T11:58:12.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff90ff9700 20 client.104438.objecter dump_active .. 0 homeless 2024-09-18T11:58:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.449+0000 7eff98f32700 10 monclient: get_auth_request con 0x7eff80038670 auth_method 0 2024-09-18T11:58:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.453+0000 7eff90ff9700 10 client.104438.objecter ms_handle_connect 0x7eff80038670 2024-09-18T11:58:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.461+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7eff8c002910 con 0x7eff941385b0 2024-09-18T11:58:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.461+0000 7eff90ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:12.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.461+0000 7eff90ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:12.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.685+0000 7eff99f34700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:58:12.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.685+0000 7eff99f34700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:12.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.685+0000 7eff99f34700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:12.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.685+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7eff740058e0 con 0x7eff941385b0 2024-09-18T11:58:12.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.685+0000 7eff90ff9700 1 -- 172.21.15.112:0/3978082397 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2273122642) 0x7eff8c014410 con 0x7eff941385b0 2024-09-18T11:58:12.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.685+0000 7eff90ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:12.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.685+0000 7eff90ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:12.690 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:58:12.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 10 librados: watch_flush enter 2024-09-18T11:58:12.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 10 librados: watch_flush exit 2024-09-18T11:58:12.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 20 client.104438.objecter shutdown clearing up homeless session... 2024-09-18T11:58:12.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 10 client.104438.objecter successfully canceled tick 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 >> v1:172.21.15.129:6812/20806 conn(0x7eff80038670 legacy=0x7eff8003ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 10 monclient: shutdown 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 >> v1:172.21.15.112:6790/0 conn(0x7eff941385b0 legacy=0x7eff94149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 shutdown_connections 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 >> 172.21.15.112:0/3978082397 conn(0x7eff940a5990 msgr2=0x7eff94138b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 shutdown_connections 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 1 -- 172.21.15.112:0/3978082397 wait complete. 2024-09-18T11:58:12.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.689+0000 7eff99f34700 1 librados: shutdown 2024-09-18T11:58:12.707 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:58:12.707 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T11:58:12.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:58:12.707 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.809+0000 7f54ecb0f700 10 monclient: get_monmap_and_config 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.809+0000 7f54ecb0f700 10 monclient: build_initial_monmap 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.809+0000 7f54ecb0f700 10 monclient: monmap: 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:12.815995+0000 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:12.815995+0000 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:12.815 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.816 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:12.816 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:12.816 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:12.816 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 1 Processor -- start 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 1 -- start start 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 10 monclient: init 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 10 monclient(hunting): picked mon.noname-a con 0x7f54e80a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 10 monclient(hunting): picked mon.noname-b con 0x7f54e806c280 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 10 monclient(hunting): picked mon.noname-c con 0x7f54e806c670 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:12.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e806e950 con 0x7f54e80a1ea0 2024-09-18T11:58:12.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e80a2270 con 0x7f54e806c670 2024-09-18T11:58:12.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e806d130 con 0x7f54e806c280 2024-09-18T11:58:12.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:12.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54ecb0f700 10 monclient(hunting): authenticate will time out at 1664.656819s 2024-09-18T11:58:12.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e6ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f54e806c280 0x7f54e8135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47768/0 (socket says 172.21.15.112:47768) 2024-09-18T11:58:12.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e6ffd700 1 -- 172.21.15.112:0/363807383 learned_addr learned my addr 172.21.15.112:0/363807383 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:12.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f54e80a1ea0 0x7f54e806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35082/0 (socket says 172.21.15.112:35082) 2024-09-18T11:58:12.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e6ffd700 1 -- 172.21.15.112:0/363807383 >> v1:172.21.15.112:6790/0 conn(0x7f54e806c280 legacy=0x7f54e8135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e77fe700 1 -- 172.21.15.112:0/363807383 >> v1:172.21.15.112:6789/0 conn(0x7f54e80a1ea0 legacy=0x7f54e806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622546033 0 0) 0x7f54e806e950 con 0x7f54e80a1ea0 2024-09-18T11:58:12.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e7fff700 1 -- 172.21.15.112:0/363807383 >> v1:172.21.15.129:6789/0 conn(0x7f54e806c670 legacy=0x7f54e81388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): my global_id is 134133 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d0003570 con 0x7f54e80a1ea0 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171018371 0 0) 0x7f54e806d130 con 0x7f54e806c280 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): my global_id is 104441 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e806e950 con 0x7f54e806c280 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447571354 0 0) 0x7f54e80a2270 con 0x7f54e806c670 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 10 monclient(hunting): my global_id is 114331 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e806d130 con 0x7f54e806c670 2024-09-18T11:58:12.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1094714566 0 0) 0x7f54d0003570 con 0x7f54e80a1ea0 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 >> v1:172.21.15.112:6790/0 conn(0x7f54e806c280 legacy=0x7f54e8135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.813+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 >> v1:172.21.15.129:6789/0 conn(0x7f54e806c670 legacy=0x7f54e81388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: _finish_hunting 0 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 1 monclient: found mon.noname-a 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e809e370 con 0x7f54e80a1ea0 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: _finish_auth 0 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:12.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f54dc001940 con 0x7f54e80a1ea0 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: dump: 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.827 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 5 monclient: authenticate success, global_id 134133 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54dc001bd0 con 0x7f54e80a1ea0 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 1 -- 172.21.15.112:0/363807383 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f54dc0049d0 con 0x7f54e80a1ea0 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:12.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e67fc700 10 monclient: dump: 2024-09-18T11:58:12.829 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.829 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.829 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.829 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.829 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.829 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.829 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.829 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.830 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.830 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:12.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 4 set_mon_vals no callback set 2024-09-18T11:58:12.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient: shutdown 2024-09-18T11:58:12.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:12.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- 172.21.15.112:0/363807383 >> v1:172.21.15.112:6789/0 conn(0x7f54e80a1ea0 legacy=0x7f54e806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- 172.21.15.112:0/363807383 shutdown_connections 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- 172.21.15.112:0/363807383 >> 172.21.15.112:0/363807383 conn(0x7f54e8067da0 msgr2=0x7f54e80a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- 172.21.15.112:0/363807383 shutdown_connections 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- 172.21.15.112:0/363807383 wait complete. 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient: build_initial_monmap 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient: monmap: 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:12.831 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:12.832 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:12.832 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.832 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:12.832 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:12.832 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:12.832 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 librados: starting msgr at 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 librados: starting objecter 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 Processor -- start 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- start start 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 librados: setting wanted keys 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 librados: calling monclient init 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient: init 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:58:12.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient(hunting): picked mon.noname-b con 0x7f54e806c280 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:12.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient(hunting): picked mon.noname-a con 0x7f54e806c670 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:12.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient(hunting): picked mon.noname-c con 0x7f54e80a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:12.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e811d020 con 0x7f54e806c670 2024-09-18T11:58:12.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e811dc90 con 0x7f54e806c280 2024-09-18T11:58:12.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e806e1a0 con 0x7f54e80a1ea0 2024-09-18T11:58:12.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:12.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54ecb0f700 10 monclient(hunting): authenticate will time out at 1664.663468s 2024-09-18T11:58:12.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e77fe700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f54e806c280 0x7f54e813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.112:36316/0 (socket says 172.21.15.112:36316) 2024-09-18T11:58:12.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.817+0000 7f54e77fe700 1 -- 172.21.15.112:0/2775823702 learned_addr learned my addr 172.21.15.112:0/2775823702 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:12.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54e77fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.129:6789/0 conn(0x7f54e806c280 legacy=0x7f54e813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 client.?.objecter ms_handle_connect 0x7f54e806c280 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 client.?.objecter ms_handle_connect 0x7f54e80a1ea0 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 client.?.objecter ms_handle_connect 0x7f54e806c670 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54e7fff700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6790/0 conn(0x7f54e80a1ea0 legacy=0x7f54e81476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54e6ffd700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6789/0 conn(0x7f54e806c670 legacy=0x7f54e81310d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54e77fe700 1 --1- 172.21.15.112:0/2775823702 >> v1:172.21.15.129:6789/0 conn(0x7f54e806c280 0x7f54e813f3a0 :-1 s=OPENED pgs=9182 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f54e811dc90 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204693267 0 0) 0x7f54e811d020 con 0x7f54e806c670 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): my global_id is 134136 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d4004f10 con 0x7f54e806c670 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673153733 0 0) 0x7f54e806e1a0 con 0x7f54e80a1ea0 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): my global_id is 104444 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e811d020 con 0x7f54e80a1ea0 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2412613218 0 0) 0x7f54d4004f10 con 0x7f54e806c670 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54e806e1a0 con 0x7f54e806c670 2024-09-18T11:58:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3852791138 0 0) 0x7f54e811d020 con 0x7f54e80a1ea0 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54d4004f10 con 0x7f54e80a1ea0 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f54e0001f40 con 0x7f54e80a1ea0 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): dump: 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.843 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f54d8001cb0 con 0x7f54e806c670 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient(hunting): dump: 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.844 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 224236962 0 0) 0x7f54e806e1a0 con 0x7f54e806c670 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6790/0 conn(0x7f54e80a1ea0 legacy=0x7f54e81476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.129:6789/0 conn(0x7f54e806c280 legacy=0x7f54e813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 monclient: found mon.a 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:12.845 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e811f200 con 0x7f54e806c670 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient: _finish_auth 0 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315735240 0 0) 0x7f54e811dc90 con 0x7f54e806c280 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 5 monclient: authenticate success, global_id 134136 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 10 monclient: _renew_subs 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e811f4e0 con 0x7f54e806c670 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T11:58:12.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 10 client.134136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54d8004be0 con 0x7f54e806c670 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 10 monclient: _renew_subs 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e813fab0 con 0x7f54e806c670 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54ecb0f700 1 librados: init done 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f54d8004ef0 con 0x7f54e806c670 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54cf7fe700 10 monclient: dump: 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.847 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.821+0000 7f54e57fa700 4 set_mon_vals no callback set 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f54d8005150 con 0x7f54e806c670 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54ecb0f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f54e805f2e0 con 0x7f54e8117c50 2024-09-18T11:58:12.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(536..536 src has 1..536) v4 ==== 5737+0+0 (unknown 449751968 0 0) 0x7f54d8043f70 con 0x7f54e806c670 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 10 client.134136.objecter ms_dispatch 0x7f54e8000fd0 osd_map(536..536 src has 1..536) v4 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 3 client.134136.objecter handle_osd_map got epochs [536,536] > 0 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 3 client.134136.objecter handle_osd_map decoding full epoch 536 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 20 client.134136.objecter dump_active .. 0 homeless 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54e7fff700 10 monclient: get_auth_request con 0x7f54d4038280 auth_method 0 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54e77fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6789/0 conn(0x7f54e8117c50 legacy=0x7f54e8118020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 10 client.134136.objecter ms_handle_connect 0x7f54e8117c50 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 10 client.134136.objecter resend_mon_ops 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129469033 0 0) 0x7f54e805f2e0 con 0x7f54e8117c50 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 10 monclient(hunting): my global_id is 134139 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e811dc90 con 0x7f54e8117c50 2024-09-18T11:58:12.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.825+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 602662100 0 0) 0x7f54e811dc90 con 0x7f54e8117c50 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54e805f2e0 con 0x7f54e8117c50 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f54dc004830 con 0x7f54e8117c50 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 10 monclient: dump: 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.850 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1125069143 0 0) 0x7f54e805f2e0 con 0x7f54e8117c50 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54e80008d0 con 0x7f54e8117c50 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 10 client.134136.objecter ms_handle_connect 0x7f54d4038280 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f54dc004a80 con 0x7f54e8117c50 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6789/0 conn(0x7f54e8117c50 legacy=0x7f54e8118020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.829+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6789/0 conn(0x7f54e8117c50 legacy=0x7f54e8118020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.837+0000 7f54ecb0f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.837+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f54e8118b00 con 0x7f54e8118730 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 client.134136.objecter ms_handle_connect 0x7f54e8118730 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 client.134136.objecter resend_mon_ops 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54e77fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6789/0 conn(0x7f54e8118730 legacy=0x7f54e802ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265600725 0 0) 0x7f54e8118b00 con 0x7f54e8118730 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 monclient(hunting): my global_id is 134142 2024-09-18T11:58:12.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e805f2e0 con 0x7f54e8118730 2024-09-18T11:58:12.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3528975556 0 0) 0x7f54e805f2e0 con 0x7f54e8118730 2024-09-18T11:58:12.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54e8118b00 con 0x7f54e8118730 2024-09-18T11:58:12.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f54dc0050a0 con 0x7f54e8118730 2024-09-18T11:58:12.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:12.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 monclient: dump: 2024-09-18T11:58:12.854 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2412643763 0 0) 0x7f54e8118b00 con 0x7f54e8118730 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f54e81317e0 con 0x7f54e8118730 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 245541412) 0x7f54dc005370 con 0x7f54e8118730 2024-09-18T11:58:12.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:58:12.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:12.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6789/0 conn(0x7f54e8118730 legacy=0x7f54e802ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54cf7fe700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6789/0 conn(0x7f54e8118730 legacy=0x7f54e802ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:12.856 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:58:12.856 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 60, 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:12.857 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:12.858 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:12.860 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:12.861 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:58:12.862 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:12.863 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:58:12.866 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:12.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:58:12.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:12.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:12.870 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:58:12.870 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:12.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.841+0000 7f54ecb0f700 10 librados: watch_flush enter 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 10 librados: watch_flush exit 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 20 client.134136.objecter shutdown clearing up homeless session... 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 10 client.134136.objecter successfully canceled tick 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.129:6812/20806 conn(0x7f54d4038280 legacy=0x7f54d403a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 10 monclient: shutdown 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 >> v1:172.21.15.112:6789/0 conn(0x7f54e806c670 legacy=0x7f54e81310d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54e7fff700 1 -- 172.21.15.112:0/2775823702 reap_dead start 2024-09-18T11:58:12.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54e7fff700 1 -- 172.21.15.112:0/2775823702 reap_dead start 2024-09-18T11:58:12.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 shutdown_connections 2024-09-18T11:58:12.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 >> 172.21.15.112:0/2775823702 conn(0x7f54e8067da0 msgr2=0x7f54e809f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:12.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 shutdown_connections 2024-09-18T11:58:12.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 1 -- 172.21.15.112:0/2775823702 wait complete. 2024-09-18T11:58:12.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.845+0000 7f54ecb0f700 1 librados: shutdown 2024-09-18T11:58:12.873 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:58:12.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:58:12.873 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:58:12.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.977+0000 7fb489751700 10 monclient: get_monmap_and_config 2024-09-18T11:58:12.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.977+0000 7fb489751700 10 monclient: build_initial_monmap 2024-09-18T11:58:12.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.977+0000 7fb489751700 10 monclient: monmap: 2024-09-18T11:58:12.983 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:12.983 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.983 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:12.984188+0000 2024-09-18T11:58:12.983 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:12.984188+0000 2024-09-18T11:58:12.984 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:12.984 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.984 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:12.984 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:12.984 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:12.984 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 1 Processor -- start 2024-09-18T11:58:12.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 1 -- start start 2024-09-18T11:58:12.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 10 monclient: init 2024-09-18T11:58:12.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:12.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:58:12.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 10 monclient(hunting): picked mon.noname-b con 0x7fb4840a3130 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:12.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 10 monclient(hunting): picked mon.noname-a con 0x7fb48409b810 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 10 monclient(hunting): picked mon.noname-c con 0x7fb48409bbe0 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48409b110 con 0x7fb48409b810 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48409ae70 con 0x7fb48409bbe0 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48409ed80 con 0x7fb4840a3130 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb489751700 10 monclient(hunting): authenticate will time out at 1664.824990s 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb4837fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb48409b810 0x7fb4841370a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35128/0 (socket says 172.21.15.112:35128) 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb4837fe700 1 -- 172.21.15.112:0/1220280334 learned_addr learned my addr 172.21.15.112:0/1220280334 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:12.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb483fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb4840a3130 0x7fb48409c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47784/0 (socket says 172.21.15.112:47784) 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb4837fe700 1 -- 172.21.15.112:0/1220280334 >> v1:172.21.15.112:6789/0 conn(0x7fb48409b810 legacy=0x7fb4841370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb483fff700 1 -- 172.21.15.112:0/1220280334 >> v1:172.21.15.112:6790/0 conn(0x7fb4840a3130 legacy=0x7fb48409c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604957528 0 0) 0x7fb48409b110 con 0x7fb48409b810 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 10 monclient(hunting): my global_id is 134145 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb488f50700 1 -- 172.21.15.112:0/1220280334 >> v1:172.21.15.129:6789/0 conn(0x7fb48409bbe0 legacy=0x7fb48413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb470003570 con 0x7fb48409b810 2024-09-18T11:58:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570619752 0 0) 0x7fb48409ed80 con 0x7fb4840a3130 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 10 monclient(hunting): my global_id is 104447 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48409b110 con 0x7fb4840a3130 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.981+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025145696 0 0) 0x7fb48409ae70 con 0x7fb48409bbe0 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient(hunting): my global_id is 114337 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48409ed80 con 0x7fb48409bbe0 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1238471765 0 0) 0x7fb470003570 con 0x7fb48409b810 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 >> v1:172.21.15.112:6790/0 conn(0x7fb4840a3130 legacy=0x7fb48409c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 >> v1:172.21.15.129:6789/0 conn(0x7fb48409bbe0 legacy=0x7fb48413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 monclient: found mon.noname-a 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4840a00d0 con 0x7fb48409b810 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: _finish_auth 0 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb46c001940 con 0x7fb48409b810 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 5 monclient: authenticate success, global_id 134145 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: dump: 2024-09-18T11:58:12.991 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb46c001bd0 con 0x7fb48409b810 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:12.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 1 -- 172.21.15.112:0/1220280334 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb46c004a90 con 0x7fb48409b810 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 4 set_mon_vals no callback set 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb482ffd700 10 monclient: dump: 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient: shutdown 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- 172.21.15.112:0/1220280334 >> v1:172.21.15.112:6789/0 conn(0x7fb48409b810 legacy=0x7fb4841370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- 172.21.15.112:0/1220280334 shutdown_connections 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- 172.21.15.112:0/1220280334 >> 172.21.15.112:0/1220280334 conn(0x7fb484069d80 msgr2=0x7fb4840a2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- 172.21.15.112:0/1220280334 shutdown_connections 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- 172.21.15.112:0/1220280334 wait complete. 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient: build_initial_monmap 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient: monmap: 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:12.994 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 librados: starting msgr at 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 librados: starting objecter 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 Processor -- start 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- start start 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 librados: setting wanted keys 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 librados: calling monclient init 2024-09-18T11:58:12.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient: init 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient(hunting): picked mon.noname-b con 0x7fb48409b810 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient(hunting): picked mon.noname-a con 0x7fb48409bbe0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient(hunting): picked mon.noname-c con 0x7fb4840a3130 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48411b130 con 0x7fb48409bbe0 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4840a1b60 con 0x7fb48409b810 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48409fc20 con 0x7fb4840a3130 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb4837fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb48409bbe0 0x7fb484146ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35132/0 (socket says 172.21.15.112:35132) 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb4837fe700 1 -- 172.21.15.112:0/873383381 learned_addr learned my addr 172.21.15.112:0/873383381 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.985+0000 7fb489751700 10 monclient(hunting): authenticate will time out at 1664.831046s 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb488f50700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6790/0 conn(0x7fb4840a3130 legacy=0x7fb48414a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 client.?.objecter ms_handle_connect 0x7fb4840a3130 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 client.?.objecter ms_handle_connect 0x7fb48409bbe0 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb4837fe700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6789/0 conn(0x7fb48409bbe0 legacy=0x7fb484146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 client.?.objecter ms_handle_connect 0x7fb48409b810 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb483fff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.129:6789/0 conn(0x7fb48409b810 legacy=0x7fb484141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349761075 0 0) 0x7fb48411b130 con 0x7fb48409bbe0 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): my global_id is 134148 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb464004f10 con 0x7fb48409bbe0 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065552807 0 0) 0x7fb48409fc20 con 0x7fb4840a3130 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): my global_id is 104450 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48411b130 con 0x7fb4840a3130 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822177507 0 0) 0x7fb4840a1b60 con 0x7fb48409b810 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): my global_id is 114340 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48409fc20 con 0x7fb48409b810 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3976947315 0 0) 0x7fb464004f10 con 0x7fb48409bbe0 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4840a1b60 con 0x7fb48409bbe0 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb46c0049d0 con 0x7fb48409bbe0 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): dump: 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.004 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3173626914 0 0) 0x7fb48411b130 con 0x7fb4840a3130 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb464004f10 con 0x7fb4840a3130 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3452655854 0 0) 0x7fb48409fc20 con 0x7fb48409b810 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48411b130 con 0x7fb48409b810 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb47c001f80 con 0x7fb4840a3130 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): dump: 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb474001b70 con 0x7fb48409b810 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient(hunting): dump: 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2329150077 0 0) 0x7fb4840a1b60 con 0x7fb48409bbe0 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6790/0 conn(0x7fb4840a3130 legacy=0x7fb48414a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.129:6789/0 conn(0x7fb48409b810 legacy=0x7fb484141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient: _finish_hunting 0 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 monclient: found mon.a 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48411abb0 con 0x7fb48409bbe0 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient: _finish_auth 0 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 5 monclient: authenticate success, global_id 134148 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 10 monclient: _renew_subs 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb484142560 con 0x7fb48409bbe0 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 10 client.134148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:13.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 10 monclient: _renew_subs 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48411b780 con 0x7fb48409bbe0 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 1 librados: init done 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb46c001580 con 0x7fb48409bbe0 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb46c0051e0 con 0x7fb48409bbe0 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb481ffb700 4 set_mon_vals no callback set 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 10 monclient: dump: 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.009 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb468006590 con 0x7fb468002840 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.989+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb46c00fa60 con 0x7fb48409bbe0 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(537..537 src has 1..537) v4 ==== 5750+0+0 (unknown 2527072992 0 0) 0x7fb46c0441e0 con 0x7fb48409bbe0 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 client.134148.objecter ms_dispatch 0x7fb484000fd0 osd_map(537..537 src has 1..537) v4 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 3 client.134148.objecter handle_osd_map got epochs [537,537] > 0 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 3 client.134148.objecter handle_osd_map decoding full epoch 537 2024-09-18T11:58:13.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 20 client.134148.objecter dump_active .. 0 homeless 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb488f50700 10 monclient: get_auth_request con 0x7fb464038650 auth_method 0 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 client.134148.objecter ms_handle_connect 0x7fb468002840 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb483fff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6790/0 conn(0x7fb468002840 legacy=0x7fb468004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 client.134148.objecter resend_mon_ops 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362599943 0 0) 0x7fb468006590 con 0x7fb468002840 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 monclient(hunting): my global_id is 104453 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4840a1b60 con 0x7fb468002840 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1546824110 0 0) 0x7fb4840a1b60 con 0x7fb468002840 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb468006590 con 0x7fb468002840 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb474004bc0 con 0x7fb468002840 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 monclient: dump: 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1987292595 0 0) 0x7fb468006590 con 0x7fb468002840 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb46800a140 con 0x7fb468002840 2024-09-18T11:58:13.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 client.134148.objecter ms_handle_connect 0x7fb464038650 2024-09-18T11:58:13.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fb474006fc0 con 0x7fb468002840 2024-09-18T11:58:13.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:13.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6790/0 conn(0x7fb468002840 legacy=0x7fb468004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:12.993+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6790/0 conn(0x7fb468002840 legacy=0x7fb468004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:13.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.001+0000 7fb489751700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:58:13.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.001+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb468009670 con 0x7fb4680090e0 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb483fff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6790/0 conn(0x7fb4680090e0 legacy=0x7fb468008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 client.134148.objecter ms_handle_connect 0x7fb4680090e0 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 client.134148.objecter resend_mon_ops 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180730109 0 0) 0x7fb468009670 con 0x7fb4680090e0 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 monclient(hunting): my global_id is 104456 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb468006590 con 0x7fb4680090e0 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1659332234 0 0) 0x7fb468006590 con 0x7fb4680090e0 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb468009670 con 0x7fb4680090e0 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb474005490 con 0x7fb4680090e0 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 monclient: dump: 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3965064274 0 0) 0x7fb468009670 con 0x7fb4680090e0 2024-09-18T11:58:13.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb468009930 con 0x7fb4680090e0 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 2025004686) 0x7fb4740067d0 con 0x7fb4680090e0 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6790/0 conn(0x7fb4680090e0 legacy=0x7fb468008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.005+0000 7fb47bfff700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6790/0 conn(0x7fb4680090e0 legacy=0x7fb468008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:58:13.017 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 60, 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:58:13.026 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.027 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:58:13.028 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 10 librados: watch_flush enter 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 10 librados: watch_flush exit 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 20 client.134148.objecter shutdown clearing up homeless session... 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 10 client.134148.objecter successfully canceled tick 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.129:6812/20806 conn(0x7fb464038650 legacy=0x7fb46403ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 10 monclient: shutdown 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 >> v1:172.21.15.112:6789/0 conn(0x7fb48409bbe0 legacy=0x7fb484146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb488f50700 1 -- 172.21.15.112:0/873383381 reap_dead start 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 shutdown_connections 2024-09-18T11:58:13.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 >> 172.21.15.112:0/873383381 conn(0x7fb484069d80 msgr2=0x7fb48409d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:13.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 shutdown_connections 2024-09-18T11:58:13.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 1 -- 172.21.15.112:0/873383381 wait complete. 2024-09-18T11:58:13.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.009+0000 7fb489751700 1 librados: shutdown 2024-09-18T11:58:13.031 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:58:13.031 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:58:13.031 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:58:13.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.137+0000 7f18a684c700 10 monclient: get_monmap_and_config 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.137+0000 7f18a684c700 10 monclient: build_initial_monmap 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.137+0000 7f18a684c700 10 monclient: monmap: 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:13.143691+0000 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:13.143691+0000 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.143 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:13.144 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:13.144 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:13.144 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.137+0000 7f18a684c700 1 Processor -- start 2024-09-18T11:58:13.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 1 -- start start 2024-09-18T11:58:13.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 10 monclient: init 2024-09-18T11:58:13.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:13.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:58:13.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 10 monclient(hunting): picked mon.noname-c con 0x7f18a00a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 10 monclient(hunting): picked mon.noname-b con 0x7f18a00a0750 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 10 monclient(hunting): picked mon.noname-a con 0x7f18a01383a0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a0096f90 con 0x7f18a01383a0 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a00a1d90 con 0x7f18a00a0380 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a0096cb0 con 0x7f18a00a0750 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 10 monclient(hunting): authenticate will time out at 1664.984632s 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a5049700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f18a00a0750 0x7f18a00a0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47814/0 (socket says 172.21.15.112:47814) 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a5049700 1 -- 172.21.15.112:0/116825947 learned_addr learned my addr 172.21.15.112:0/116825947 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:13.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a604b700 1 --1- 172.21.15.112:0/116825947 >> v1:172.21.15.112:6789/0 conn(0x7f18a01383a0 0x7f18a013a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35144/0 (socket says 172.21.15.112:35144) 2024-09-18T11:58:13.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a5049700 1 -- 172.21.15.112:0/116825947 >> v1:172.21.15.112:6790/0 conn(0x7f18a00a0750 legacy=0x7f18a00a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a604b700 1 -- 172.21.15.112:0/116825947 >> v1:172.21.15.112:6789/0 conn(0x7f18a01383a0 legacy=0x7f18a013a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197156694 0 0) 0x7f18a0096cb0 con 0x7f18a00a0750 2024-09-18T11:58:13.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient(hunting): my global_id is 104459 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f188c003570 con 0x7f18a00a0750 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a584a700 1 -- 172.21.15.112:0/116825947 >> v1:172.21.15.129:6789/0 conn(0x7f18a00a0380 legacy=0x7f18a0097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298555890 0 0) 0x7f18a0096f90 con 0x7f18a01383a0 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient(hunting): my global_id is 134151 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a0096cb0 con 0x7f18a01383a0 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 758798819 0 0) 0x7f188c003570 con 0x7f18a00a0750 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 >> v1:172.21.15.129:6789/0 conn(0x7f18a00a0380 legacy=0x7f18a0097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 >> v1:172.21.15.112:6789/0 conn(0x7f18a01383a0 legacy=0x7f18a013a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient: _finish_hunting 0 2024-09-18T11:58:13.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 monclient: found mon.noname-b 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0097270 con 0x7f18a00a0750 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient: _finish_auth 0 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 5 monclient: authenticate success, global_id 104459 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1890001940 con 0x7f18a00a0750 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:58:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a4848700 10 monclient: dump: 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.141+0000 7f18a684c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1890001bd0 con 0x7f18a00a0750 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a4848700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:13.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 4 set_mon_vals no callback set 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a4848700 1 -- 172.21.15.112:0/116825947 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1890004a90 con 0x7f18a00a0750 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a4848700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a4848700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a4848700 10 monclient: dump: 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.152 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.153 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient: shutdown 2024-09-18T11:58:13.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:13.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- 172.21.15.112:0/116825947 >> v1:172.21.15.112:6790/0 conn(0x7f18a00a0750 legacy=0x7f18a00a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- 172.21.15.112:0/116825947 shutdown_connections 2024-09-18T11:58:13.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- 172.21.15.112:0/116825947 >> 172.21.15.112:0/116825947 conn(0x7f18a00a6410 msgr2=0x7f18a00a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:13.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- 172.21.15.112:0/116825947 shutdown_connections 2024-09-18T11:58:13.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- 172.21.15.112:0/116825947 wait complete. 2024-09-18T11:58:13.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient: build_initial_monmap 2024-09-18T11:58:13.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient: monmap: 2024-09-18T11:58:13.157 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:13.157 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:13.157 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:13.157 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 librados: starting msgr at 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 librados: starting objecter 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 Processor -- start 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- start start 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 librados: setting wanted keys 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 librados: calling monclient init 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient: init 2024-09-18T11:58:13.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient(hunting): picked mon.noname-b con 0x7f18a00a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient(hunting): picked mon.noname-a con 0x7f18a00a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient(hunting): picked mon.noname-c con 0x7f18a01383a0 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a009bb80 con 0x7f18a00a0750 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a009be60 con 0x7f18a00a0380 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a009c140 con 0x7f18a01383a0 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a684c700 10 monclient(hunting): authenticate will time out at 1664.991240s 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a5049700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f18a00a0750 0x7f18a009e960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35146/0 (socket says 172.21.15.112:35146) 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.145+0000 7f18a5049700 1 -- 172.21.15.112:0/2395037872 learned_addr learned my addr 172.21.15.112:0/2395037872 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a604b700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.112:6790/0 conn(0x7f18a01383a0 legacy=0x7f18a0149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 client.?.objecter ms_handle_connect 0x7f18a01383a0 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 client.?.objecter ms_handle_connect 0x7f18a00a0750 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a5049700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.112:6789/0 conn(0x7f18a00a0750 legacy=0x7f18a009e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 client.?.objecter ms_handle_connect 0x7f18a00a0380 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a584a700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6789/0 conn(0x7f18a00a0380 legacy=0x7f18a0141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091275433 0 0) 0x7f18a009c140 con 0x7f18a01383a0 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): my global_id is 104462 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1884004f10 con 0x7f18a01383a0 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177482428 0 0) 0x7f18a009be60 con 0x7f18a00a0380 2024-09-18T11:58:13.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): my global_id is 114346 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a009c140 con 0x7f18a00a0380 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426121242 0 0) 0x7f18a009bb80 con 0x7f18a00a0750 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): my global_id is 134154 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a009be60 con 0x7f18a00a0750 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439358722 0 0) 0x7f18a009be60 con 0x7f18a00a0750 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18a009bb80 con 0x7f18a00a0750 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2279239830 0 0) 0x7f1884004f10 con 0x7f18a01383a0 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18a009be60 con 0x7f18a01383a0 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1890004830 con 0x7f18a00a0750 2024-09-18T11:58:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): dump: 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1898001fb0 con 0x7f18a01383a0 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): dump: 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.163 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2974887750 0 0) 0x7f18a009c140 con 0x7f18a00a0380 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1884004f10 con 0x7f18a00a0380 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f189c001ae0 con 0x7f18a00a0380 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient(hunting): dump: 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.164 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 7034179 0 0) 0x7f18a009bb80 con 0x7f18a00a0750 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.112:6790/0 conn(0x7f18a01383a0 legacy=0x7f18a0149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6789/0 conn(0x7f18a00a0380 legacy=0x7f18a0141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 monclient: found mon.a 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a009ac90 con 0x7f18a00a0750 2024-09-18T11:58:13.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient: _finish_auth 0 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 5 monclient: authenticate success, global_id 134154 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 10 monclient: _renew_subs 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a0141a90 con 0x7f18a00a0750 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 10 client.134154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 10 monclient: _renew_subs 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a009f210 con 0x7f18a00a0750 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f18a684c700 1 librados: init done 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1890004ad0 con 0x7f18a00a0750 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1890004fc0 con 0x7f18a00a0750 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1895ffb700 10 monclient: dump: 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.149+0000 7f1897fff700 4 set_mon_vals no callback set 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f18a684c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f189000f900 con 0x7f18a00a0750 2024-09-18T11:58:13.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1880006590 con 0x7f1880002840 2024-09-18T11:58:13.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(537..537 src has 1..537) v4 ==== 5750+0+0 (unknown 2527072992 0 0) 0x7f1890043fd0 con 0x7f18a00a0750 2024-09-18T11:58:13.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 10 client.134154.objecter ms_dispatch 0x7f18a0000fd0 osd_map(537..537 src has 1..537) v4 2024-09-18T11:58:13.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 3 client.134154.objecter handle_osd_map got epochs [537,537] > 0 2024-09-18T11:58:13.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 3 client.134154.objecter handle_osd_map decoding full epoch 537 2024-09-18T11:58:13.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 20 client.134154.objecter dump_active .. 0 homeless 2024-09-18T11:58:13.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f18a604b700 10 monclient: get_auth_request con 0x7f1884038540 auth_method 0 2024-09-18T11:58:13.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f18a584a700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6789/0 conn(0x7f1880002840 legacy=0x7f1880004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 10 client.134154.objecter ms_handle_connect 0x7f1880002840 2024-09-18T11:58:13.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 10 client.134154.objecter resend_mon_ops 2024-09-18T11:58:13.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 10 client.134154.objecter ms_handle_connect 0x7f1884038540 2024-09-18T11:58:13.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149630785 0 0) 0x7f1880006590 con 0x7f1880002840 2024-09-18T11:58:13.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 10 monclient(hunting): my global_id is 114349 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.153+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a009bb80 con 0x7f1880002840 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1261588963 0 0) 0x7f18a009bb80 con 0x7f1880002840 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1880006590 con 0x7f1880002840 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f189c002310 con 0x7f1880002840 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 10 monclient: dump: 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.170 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2296787241 0 0) 0x7f1880006590 con 0x7f1880002840 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f188000a140 con 0x7f1880002840 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f189c005810 con 0x7f1880002840 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6789/0 conn(0x7f1880002840 legacy=0x7f1880004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.157+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6789/0 conn(0x7f1880002840 legacy=0x7f1880004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:13.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f18a684c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:58:13.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1880009f10 con 0x7f1880006900 2024-09-18T11:58:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f18a584a700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6789/0 conn(0x7f1880006900 legacy=0x7f1880006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f1895ffb700 10 client.134154.objecter ms_handle_connect 0x7f1880006900 2024-09-18T11:58:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f1895ffb700 10 client.134154.objecter resend_mon_ops 2024-09-18T11:58:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117204399 0 0) 0x7f1880009f10 con 0x7f1880006900 2024-09-18T11:58:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f1895ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f1895ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f1895ffb700 10 monclient(hunting): my global_id is 114352 2024-09-18T11:58:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.169+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1880006590 con 0x7f1880006900 2024-09-18T11:58:13.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1081090139 0 0) 0x7f1880006590 con 0x7f1880006900 2024-09-18T11:58:13.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1880009f10 con 0x7f1880006900 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f189c002900 con 0x7f1880006900 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 10 monclient: dump: 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2838387981 0 0) 0x7f1880009f10 con 0x7f1880006900 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1880008760 con 0x7f1880006900 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 2997620832) 0x7f189c002ae0 con 0x7f1880006900 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6789/0 conn(0x7f1880006900 legacy=0x7f1880006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f1895ffb700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6789/0 conn(0x7f1880006900 legacy=0x7f1880006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:13.178 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 60, 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:13.179 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:13.180 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:58:13.181 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:13.182 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:13.183 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:13.184 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:13.186 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:58:13.187 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.188 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:13.189 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:13.190 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:58:13.190 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 10 librados: watch_flush enter 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 10 librados: watch_flush exit 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 20 client.134154.objecter shutdown clearing up homeless session... 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 10 client.134154.objecter successfully canceled tick 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.129:6812/20806 conn(0x7f1884038540 legacy=0x7f188403a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 10 monclient: shutdown 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 >> v1:172.21.15.112:6789/0 conn(0x7f18a00a0750 legacy=0x7f18a009e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a604b700 1 -- 172.21.15.112:0/2395037872 reap_dead start 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 shutdown_connections 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 >> 172.21.15.112:0/2395037872 conn(0x7f18a00a6410 msgr2=0x7f18a00a3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 shutdown_connections 2024-09-18T11:58:13.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 1 -- 172.21.15.112:0/2395037872 wait complete. 2024-09-18T11:58:13.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.173+0000 7f18a684c700 1 librados: shutdown 2024-09-18T11:58:13.193 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:58:13.193 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T11:58:13.193 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T11:58:13.193 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T11:58:13.193 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-18T11:58:13.193 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-18T11:58:13.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.305+0000 7f7bf35f3700 10 monclient: get_monmap_and_config 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.305+0000 7f7bf35f3700 10 monclient: build_initial_monmap 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.305+0000 7f7bf35f3700 10 monclient: monmap: 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:13.310619+0000 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:13.310619+0000 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:13.310 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.305+0000 7f7bf35f3700 1 Processor -- start 2024-09-18T11:58:13.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.305+0000 7f7bf35f3700 1 -- start start 2024-09-18T11:58:13.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.305+0000 7f7bf35f3700 10 monclient: init 2024-09-18T11:58:13.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:13.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:58:13.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 10 monclient(hunting): picked mon.noname-b con 0x7f7bec0a0500 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:13.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 10 monclient(hunting): picked mon.noname-c con 0x7f7bec0a1970 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:13.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 10 monclient(hunting): picked mon.noname-a con 0x7f7bec0a0fe0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:13.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec0970b0 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec0a33f0 con 0x7f7bec0a1970 2024-09-18T11:58:13.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec096dd0 con 0x7f7bec0a0500 2024-09-18T11:58:13.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:13.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf35f3700 10 monclient(hunting): authenticate will time out at 1665.152542s 2024-09-18T11:58:13.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf1b90700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f7bec0a0fe0 0x7f7bec13aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35156/0 (socket says 172.21.15.112:35156) 2024-09-18T11:58:13.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf1b90700 1 -- 172.21.15.112:0/2590493403 learned_addr learned my addr 172.21.15.112:0/2590493403 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:13.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf1b90700 1 -- 172.21.15.112:0/2590493403 >> v1:172.21.15.112:6789/0 conn(0x7f7bec0a0fe0 legacy=0x7f7bec13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf138f700 1 -- 172.21.15.112:0/2590493403 >> v1:172.21.15.112:6790/0 conn(0x7f7bec0a0500 legacy=0x7f7bec0a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bf0b8e700 1 -- 172.21.15.112:0/2590493403 >> v1:172.21.15.129:6789/0 conn(0x7f7bec0a1970 legacy=0x7f7bec1377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793462842 0 0) 0x7f7bec0970b0 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): my global_id is 134157 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bd8003570 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899331067 0 0) 0x7f7bec0a33f0 con 0x7f7bec0a1970 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): my global_id is 114355 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec0970b0 con 0x7f7bec0a1970 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135286903 0 0) 0x7f7bec096dd0 con 0x7f7bec0a0500 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 10 monclient(hunting): my global_id is 104465 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec0a33f0 con 0x7f7bec0a0500 2024-09-18T11:58:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.309+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4099559077 0 0) 0x7f7bec0970b0 con 0x7f7bec0a1970 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 >> v1:172.21.15.112:6790/0 conn(0x7f7bec0a0500 legacy=0x7f7bec0a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 >> v1:172.21.15.112:6789/0 conn(0x7f7bec0a0fe0 legacy=0x7f7bec13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: _finish_hunting 0 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 1 monclient: found mon.noname-c 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec097390 con 0x7f7bec0a1970 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: _finish_auth 0 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 5 monclient: authenticate success, global_id 114355 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7be8001970 con 0x7f7bec0a1970 2024-09-18T11:58:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: dump: 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7be8001c00 con 0x7f7bec0a1970 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 4 set_mon_vals no callback set 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 1 -- 172.21.15.112:0/2590493403 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7be80020a0 con 0x7f7bec0a1970 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bdffff700 10 monclient: dump: 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient: shutdown 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- 172.21.15.112:0/2590493403 >> v1:172.21.15.129:6789/0 conn(0x7f7bec0a1970 legacy=0x7f7bec1377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- 172.21.15.112:0/2590493403 shutdown_connections 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- 172.21.15.112:0/2590493403 >> 172.21.15.112:0/2590493403 conn(0x7f7bec0a5b10 msgr2=0x7f7bec0a5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- 172.21.15.112:0/2590493403 shutdown_connections 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- 172.21.15.112:0/2590493403 wait complete. 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient: build_initial_monmap 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient: monmap: 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 librados: starting msgr at 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 librados: starting objecter 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 Processor -- start 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- start start 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 librados: setting wanted keys 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 librados: calling monclient init 2024-09-18T11:58:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient: init 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient(hunting): picked mon.noname-a con 0x7f7bec0a0500 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient(hunting): picked mon.noname-c con 0x7f7bec0a0fe0 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient(hunting): picked mon.noname-b con 0x7f7bec0a1970 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec11ffa0 con 0x7f7bec0a0500 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec0a1e50 con 0x7f7bec0a1970 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec121b10 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf35f3700 10 monclient(hunting): authenticate will time out at 1665.159102s 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf138f700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f7bec0a0500 0x7f7bec134430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35166/0 (socket says 172.21.15.112:35166) 2024-09-18T11:58:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.313+0000 7f7bf138f700 1 -- 172.21.15.112:0/1802702023 learned_addr learned my addr 172.21.15.112:0/1802702023 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf138f700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.112:6789/0 conn(0x7f7bec0a0500 legacy=0x7f7bec134430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 client.?.objecter ms_handle_connect 0x7f7bec0a0500 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 client.?.objecter ms_handle_connect 0x7f7bec0a0fe0 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf0b8e700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.112:6790/0 conn(0x7f7bec0a0fe0 legacy=0x7f7bec146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf1b90700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6789/0 conn(0x7f7bec0a1970 legacy=0x7f7bec14a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 client.?.objecter ms_handle_connect 0x7f7bec0a1970 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887592403 0 0) 0x7f7bec11ffa0 con 0x7f7bec0a0500 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): my global_id is 134160 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bd0004f10 con 0x7f7bec0a0500 2024-09-18T11:58:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452876343 0 0) 0x7f7bec121b10 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): my global_id is 104468 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec11ffa0 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118672062 0 0) 0x7f7bec0a1e50 con 0x7f7bec0a1970 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): my global_id is 114358 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec121b10 con 0x7f7bec0a1970 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987991937 0 0) 0x7f7bd0004f10 con 0x7f7bec0a0500 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bec0a1e50 con 0x7f7bec0a0500 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7be0001d80 con 0x7f7bec0a0500 2024-09-18T11:58:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): dump: 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 20474106 0 0) 0x7f7bec11ffa0 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bd0004f10 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7be80020a0 con 0x7f7bec0a0fe0 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): dump: 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973521366 0 0) 0x7f7bec121b10 con 0x7f7bec0a1970 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bec11ffa0 con 0x7f7bec0a1970 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7be4001fc0 con 0x7f7bec0a1970 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient(hunting): dump: 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4174494450 0 0) 0x7f7bec0a1e50 con 0x7f7bec0a0500 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.112:6790/0 conn(0x7f7bec0a0fe0 legacy=0x7f7bec146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6789/0 conn(0x7f7bec0a1970 legacy=0x7f7bec14a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient: _finish_hunting 0 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 monclient: found mon.a 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec121830 con 0x7f7bec0a0500 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient: _finish_auth 0 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 5 monclient: authenticate success, global_id 134160 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 10 monclient: _renew_subs 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bec1202d0 con 0x7f7bec0a0500 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 10 client.134160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 10 monclient: _renew_subs 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bec136e90 con 0x7f7bec0a0500 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bf35f3700 1 librados: init done 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7be00021b0 con 0x7f7bec0a0500 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7be0002500 con 0x7f7bec0a0500 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdeffd700 4 set_mon_vals no callback set 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.317+0000 7f7bdd7fa700 10 monclient: dump: 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bf35f3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7bcc006590 con 0x7f7bcc002840 2024-09-18T11:58:13.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f7be0002660 con 0x7f7bec0a0500 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(537..537 src has 1..537) v4 ==== 5750+0+0 (unknown 2527072992 0 0) 0x7f7be0043fa0 con 0x7f7bec0a0500 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 client.134160.objecter ms_dispatch 0x7f7bec000fd0 osd_map(537..537 src has 1..537) v4 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 3 client.134160.objecter handle_osd_map got epochs [537,537] > 0 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 3 client.134160.objecter handle_osd_map decoding full epoch 537 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 20 client.134160.objecter dump_active .. 0 homeless 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bf1b90700 10 monclient: get_auth_request con 0x7f7bd0038620 auth_method 0 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bf0b8e700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6789/0 conn(0x7f7bcc002840 legacy=0x7f7bcc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 client.134160.objecter ms_handle_connect 0x7f7bcc002840 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 client.134160.objecter resend_mon_ops 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340403287 0 0) 0x7f7bcc006590 con 0x7f7bcc002840 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 monclient(hunting): my global_id is 114361 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec0a1e50 con 0x7f7bcc002840 2024-09-18T11:58:13.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2907391316 0 0) 0x7f7bec0a1e50 con 0x7f7bcc002840 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bcc006590 con 0x7f7bcc002840 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7be80027b0 con 0x7f7bcc002840 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.321+0000 7f7bdd7fa700 10 monclient: dump: 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.339 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.325+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3648591128 0 0) 0x7f7bcc006590 con 0x7f7bcc002840 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.325+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bcc00a140 con 0x7f7bcc002840 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.325+0000 7f7bdd7fa700 10 client.134160.objecter ms_handle_connect 0x7f7bd0038620 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.325+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f7be8002590 con 0x7f7bcc002840 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.325+0000 7f7bdd7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.325+0000 7f7bdd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.325+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6789/0 conn(0x7f7bcc002840 legacy=0x7f7bcc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.325+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6789/0 conn(0x7f7bcc002840 legacy=0x7f7bcc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:13.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.337+0000 7f7bf35f3700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:58:13.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.337+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7bcc007570 con 0x7f7bcc006920 2024-09-18T11:58:13.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bf0b8e700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6789/0 conn(0x7f7bcc006920 legacy=0x7f7bcc006da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 client.134160.objecter ms_handle_connect 0x7f7bcc006920 2024-09-18T11:58:13.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 client.134160.objecter resend_mon_ops 2024-09-18T11:58:13.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451333800 0 0) 0x7f7bcc007570 con 0x7f7bcc006920 2024-09-18T11:58:13.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 monclient(hunting): my global_id is 114364 2024-09-18T11:58:13.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bcc006590 con 0x7f7bcc006920 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4230470538 0 0) 0x7f7bcc006590 con 0x7f7bcc006920 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bcc007570 con 0x7f7bcc006920 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7be8002ca0 con 0x7f7bcc006920 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 monclient: dump: 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.346 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.347 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.347 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.347 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.347 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 190083785 0 0) 0x7f7bcc007570 con 0x7f7bcc006920 2024-09-18T11:58:13.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7f7bcc008800 con 0x7f7bcc006920 2024-09-18T11:58:13.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7f7be8005a30 con 0x7f7bcc006920 2024-09-18T11:58:13.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.341+0000 7f7bdd7fa700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-18T11:58:13.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bdd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:13.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6789/0 conn(0x7f7bcc006920 legacy=0x7f7bcc006da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bdd7fa700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6789/0 conn(0x7f7bcc006920 legacy=0x7f7bcc006da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:13.348 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:13.348 INFO:teuthology.orchestra.run.smithi112.stdout: "ret": 0, 2024-09-18T11:58:13.348 INFO:teuthology.orchestra.run.smithi112.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-18T11:58:13.348 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:13.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 10 librados: watch_flush enter 2024-09-18T11:58:13.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 10 librados: watch_flush exit 2024-09-18T11:58:13.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 20 client.134160.objecter shutdown clearing up homeless session... 2024-09-18T11:58:13.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 10 client.134160.objecter successfully canceled tick 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.129:6812/20806 conn(0x7f7bd0038620 legacy=0x7f7bd003aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 10 monclient: shutdown 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 >> v1:172.21.15.112:6789/0 conn(0x7f7bec0a0500 legacy=0x7f7bec134430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf1b90700 1 -- 172.21.15.112:0/1802702023 reap_dead start 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf1b90700 1 -- 172.21.15.112:0/1802702023 reap_dead start 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 shutdown_connections 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 >> 172.21.15.112:0/1802702023 conn(0x7f7bec0a5b10 msgr2=0x7f7bec135600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 shutdown_connections 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 1 -- 172.21.15.112:0/1802702023 wait complete. 2024-09-18T11:58:13.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.345+0000 7f7bf35f3700 1 librados: shutdown 2024-09-18T11:58:13.364 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T11:58:13.364 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T11:58:13.364 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:58:13.384 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T11:58:13.384 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T11:58:13.384 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:58:13.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient: get_monmap_and_config 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient: build_initial_monmap 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient: monmap: 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:13.482594+0000 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:13.482594+0000 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:13.482 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 1 Processor -- start 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 1 -- start start 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient: init 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient(hunting): picked mon.noname-b con 0x7f352c06aa40 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient(hunting): picked mon.noname-c con 0x7f352c0a1e80 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient(hunting): picked mon.noname-a con 0x7f352c1386e0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c097270 con 0x7f352c1386e0 2024-09-18T11:58:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c06ae10 con 0x7f352c0a1e80 2024-09-18T11:58:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c097550 con 0x7f352c06aa40 2024-09-18T11:58:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f35319a1700 10 monclient(hunting): authenticate will time out at 1665.323643s 2024-09-18T11:58:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f352a7fc700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f352c0a1e80 legacy=0x7f352c0a1160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f352affd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f352c06aa40 0x7f352c0a0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47872/0 (socket says 172.21.15.112:47872) 2024-09-18T11:58:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f352b7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f352c1386e0 0x7f352c13aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35178/0 (socket says 172.21.15.112:35178) 2024-09-18T11:58:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.477+0000 7f352affd700 1 -- 172.21.15.112:0/620891602 learned_addr learned my addr 172.21.15.112:0/620891602 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:13.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f352affd700 1 -- 172.21.15.112:0/620891602 >> v1:172.21.15.112:6790/0 conn(0x7f352c06aa40 legacy=0x7f352c0a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f352b7fe700 1 -- 172.21.15.112:0/620891602 >> v1:172.21.15.112:6789/0 conn(0x7f352c1386e0 legacy=0x7f352c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294035930 0 0) 0x7f352c097550 con 0x7f352c06aa40 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient(hunting): my global_id is 104471 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3518003570 con 0x7f352c06aa40 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618663053 0 0) 0x7f352c097270 con 0x7f352c1386e0 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient(hunting): my global_id is 134163 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c097550 con 0x7f352c1386e0 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3373922810 0 0) 0x7f352c097550 con 0x7f352c1386e0 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 >> v1:172.21.15.112:6790/0 conn(0x7f352c06aa40 legacy=0x7f352c0a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 >> v1:172.21.15.129:6789/0 conn(0x7f352c0a1e80 legacy=0x7f352c0a1160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:58:13.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 monclient: found mon.noname-a 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c096a60 con 0x7f352c1386e0 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: _finish_auth 0 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f35319a1700 5 monclient: authenticate success, global_id 134163 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f35319a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f351c001910 con 0x7f352c1386e0 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:13.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: dump: 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f35319a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f351c001a70 con 0x7f352c1386e0 2024-09-18T11:58:13.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f35319a1700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f35319a1700 4 set_mon_vals no callback set 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 1 -- 172.21.15.112:0/620891602 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f351c004830 con 0x7f352c1386e0 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.481+0000 7f3529ffb700 10 monclient: dump: 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.491 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient: shutdown 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- 172.21.15.112:0/620891602 >> v1:172.21.15.112:6789/0 conn(0x7f352c1386e0 legacy=0x7f352c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- 172.21.15.112:0/620891602 shutdown_connections 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- 172.21.15.112:0/620891602 >> 172.21.15.112:0/620891602 conn(0x7f352c0a5d10 msgr2=0x7f352c0a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- 172.21.15.112:0/620891602 shutdown_connections 2024-09-18T11:58:13.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- 172.21.15.112:0/620891602 wait complete. 2024-09-18T11:58:13.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient: build_initial_monmap 2024-09-18T11:58:13.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient: monmap: 2024-09-18T11:58:13.496 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:13.496 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:13.496 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 librados: starting msgr at 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 librados: starting objecter 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 Processor -- start 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- start start 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 librados: setting wanted keys 2024-09-18T11:58:13.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 librados: calling monclient init 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient: init 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient(hunting): picked mon.noname-a con 0x7f352c06aa40 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient(hunting): picked mon.noname-b con 0x7f352c0a1e80 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient(hunting): picked mon.noname-c con 0x7f352c1386e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0a22f0 con 0x7f352c06aa40 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f352affd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f352c06aa40 0x7f352c137b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35186/0 (socket says 172.21.15.112:35186) 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c1324d0 con 0x7f352c0a1e80 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f352affd700 1 -- 172.21.15.112:0/3961290796 learned_addr learned my addr 172.21.15.112:0/3961290796 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c132e00 con 0x7f352c1386e0 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:13.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f35319a1700 10 monclient(hunting): authenticate will time out at 1665.331050s 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.485+0000 7f352a7fc700 1 -- 172.21.15.112:0/3961290796 >> v1:172.21.15.129:6789/0 conn(0x7f352c0a1e80 legacy=0x7f352c146bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f352affd700 1 -- 172.21.15.112:0/3961290796 >> v1:172.21.15.112:6789/0 conn(0x7f352c06aa40 legacy=0x7f352c137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 client.?.objecter ms_handle_connect 0x7f352c06aa40 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 client.?.objecter ms_handle_connect 0x7f352c1386e0 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f352b7fe700 1 -- 172.21.15.112:0/3961290796 >> v1:172.21.15.112:6790/0 conn(0x7f352c1386e0 legacy=0x7f352c14a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874436936 0 0) 0x7f352c0a22f0 con 0x7f352c06aa40 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): my global_id is 134166 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350c004f10 con 0x7f352c06aa40 2024-09-18T11:58:13.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870682758 0 0) 0x7f352c132e00 con 0x7f352c1386e0 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): my global_id is 104474 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c0a22f0 con 0x7f352c1386e0 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 874978760 0 0) 0x7f350c004f10 con 0x7f352c06aa40 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f352c132e00 con 0x7f352c06aa40 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3514004830 con 0x7f352c06aa40 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): dump: 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.500 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1170525932 0 0) 0x7f352c0a22f0 con 0x7f352c1386e0 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350c004f10 con 0x7f352c1386e0 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f351c001be0 con 0x7f352c1386e0 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient(hunting): dump: 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3504445233 0 0) 0x7f352c132e00 con 0x7f352c06aa40 2024-09-18T11:58:13.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 >> v1:172.21.15.112:6790/0 conn(0x7f352c1386e0 legacy=0x7f352c14a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 >> v1:172.21.15.129:6789/0 conn(0x7f352c0a1e80 legacy=0x7f352c146bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 monclient: found mon.a 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c133d60 con 0x7f352c06aa40 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient: _finish_auth 0 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 5 monclient: authenticate success, global_id 134166 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 10 monclient: _renew_subs 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352c134040 con 0x7f352c06aa40 2024-09-18T11:58:13.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 10 client.134166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 10 monclient: _renew_subs 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f352c138400 con 0x7f352c06aa40 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35319a1700 1 librados: init done 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3514004cb0 con 0x7f352c06aa40 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f3514005000 con 0x7f352c06aa40 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f3522ffd700 10 monclient: dump: 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:13.504 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.489+0000 7f35297fa700 4 set_mon_vals no callback set 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f351400f900 con 0x7f352c06aa40 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f35319a1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f35319a1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f35319a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35100050b0 con 0x7f352c06aa40 2024-09-18T11:58:13.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(537..537 src has 1..537) v4 ==== 5750+0+0 (unknown 2527072992 0 0) 0x7f35140441b0 con 0x7f352c06aa40 2024-09-18T11:58:13.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f3522ffd700 10 client.134166.objecter ms_dispatch 0x7f352c000fd0 osd_map(537..537 src has 1..537) v4 2024-09-18T11:58:13.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f3522ffd700 3 client.134166.objecter handle_osd_map got epochs [537,537] > 0 2024-09-18T11:58:13.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f3522ffd700 3 client.134166.objecter handle_osd_map decoding full epoch 537 2024-09-18T11:58:13.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f3522ffd700 20 client.134166.objecter dump_active .. 0 homeless 2024-09-18T11:58:13.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f352a7fc700 10 monclient: get_auth_request con 0x7f350c038120 auth_method 0 2024-09-18T11:58:13.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.493+0000 7f3522ffd700 10 client.134166.objecter ms_handle_connect 0x7f350c038120 2024-09-18T11:58:13.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.501+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f3514014c20 con 0x7f352c06aa40 2024-09-18T11:58:13.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.501+0000 7f3522ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:13.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.501+0000 7f3522ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:13.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.721+0000 7f35319a1700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:58:13.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.721+0000 7f35319a1700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:13.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.721+0000 7f35319a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:13.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.721+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f35100058e0 con 0x7f352c06aa40 2024-09-18T11:58:13.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f3522ffd700 1 -- 172.21.15.112:0/3961290796 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3648554186) 0x7f3514014620 con 0x7f352c06aa40 2024-09-18T11:58:13.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f3522ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:13.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f3522ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:13.728 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:58:13.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 10 librados: watch_flush enter 2024-09-18T11:58:13.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 10 librados: watch_flush exit 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 20 client.134166.objecter shutdown clearing up homeless session... 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 10 client.134166.objecter successfully canceled tick 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 >> v1:172.21.15.129:6812/20806 conn(0x7f350c038120 legacy=0x7f350c03a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 10 monclient: shutdown 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 >> v1:172.21.15.112:6789/0 conn(0x7f352c06aa40 legacy=0x7f352c137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 shutdown_connections 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 >> 172.21.15.112:0/3961290796 conn(0x7f352c0a5d10 msgr2=0x7f352c097d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 shutdown_connections 2024-09-18T11:58:13.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 1 -- 172.21.15.112:0/3961290796 wait complete. 2024-09-18T11:58:13.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:13.725+0000 7f35319a1700 1 librados: shutdown 2024-09-18T11:58:16.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:58:16.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient: get_monmap_and_config 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient: build_initial_monmap 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient: monmap: 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:16.853719+0000 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:16.853719+0000 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:16.853 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:16.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 1 Processor -- start 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 1 -- start start 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient: init 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient(hunting): picked mon.noname-a con 0x7f10b80a1230 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient(hunting): picked mon.noname-c con 0x7f10b806e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient(hunting): picked mon.noname-b con 0x7f10b8135300 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80a1600 con 0x7f10b80a1230 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b806eda0 con 0x7f10b806e9d0 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b806b5a0 con 0x7f10b8135300 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10bd7ce700 10 monclient(hunting): authenticate will time out at 1668.695187s 2024-09-18T11:58:16.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10b67fc700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f10b806e9d0 legacy=0x7f10b806ae90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:16.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10b6ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f10b80a1230 0x7f10b806a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35194/0 (socket says 172.21.15.112:35194) 2024-09-18T11:58:16.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.849+0000 7f10b6ffd700 1 -- 172.21.15.112:0/3380456767 learned_addr learned my addr 172.21.15.112:0/3380456767 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:16.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b77fe700 1 -- 172.21.15.112:0/3380456767 >> v1:172.21.15.112:6790/0 conn(0x7f10b8135300 legacy=0x7f10b81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:16.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b6ffd700 1 -- 172.21.15.112:0/3380456767 >> v1:172.21.15.112:6789/0 conn(0x7f10b80a1230 legacy=0x7f10b806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:16.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916886754 0 0) 0x7f10b80a1600 con 0x7f10b80a1230 2024-09-18T11:58:16.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:16.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:16.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient(hunting): my global_id is 134169 2024-09-18T11:58:16.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a0003570 con 0x7f10b80a1230 2024-09-18T11:58:16.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219685221 0 0) 0x7f10a0003570 con 0x7f10b80a1230 2024-09-18T11:58:16.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 >> v1:172.21.15.112:6790/0 conn(0x7f10b8135300 legacy=0x7f10b81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:16.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 >> v1:172.21.15.129:6789/0 conn(0x7f10b806e9d0 legacy=0x7f10b806ae90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 monclient: found mon.noname-a 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b809f9e0 con 0x7f10b80a1230 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: _finish_auth 0 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10bd7ce700 5 monclient: authenticate success, global_id 134169 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10bd7ce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f10a8001870 con 0x7f10b80a1230 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: dump: 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10bd7ce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10a8001b40 con 0x7f10b80a1230 2024-09-18T11:58:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 1 -- 172.21.15.112:0/3380456767 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f10a80048f0 con 0x7f10b80a1230 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10b5ffb700 10 monclient: dump: 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:16.861 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:16.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10bd7ce700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:16.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10bd7ce700 4 set_mon_vals no callback set 2024-09-18T11:58:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.853+0000 7f10bd7ce700 10 monclient: shutdown 2024-09-18T11:58:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- 172.21.15.112:0/3380456767 >> v1:172.21.15.112:6789/0 conn(0x7f10b80a1230 legacy=0x7f10b806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- 172.21.15.112:0/3380456767 shutdown_connections 2024-09-18T11:58:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- 172.21.15.112:0/3380456767 >> 172.21.15.112:0/3380456767 conn(0x7f10b80680d0 msgr2=0x7f10b80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- 172.21.15.112:0/3380456767 shutdown_connections 2024-09-18T11:58:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- 172.21.15.112:0/3380456767 wait complete. 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient: build_initial_monmap 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient: monmap: 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:16.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 librados: starting msgr at 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 librados: starting objecter 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 Processor -- start 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- start start 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 librados: setting wanted keys 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 librados: calling monclient init 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient: init 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient(hunting): picked mon.noname-a con 0x7f10b806e9d0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient(hunting): picked mon.noname-b con 0x7f10b80a1230 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient(hunting): picked mon.noname-c con 0x7f10b8135300 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b812f1e0 con 0x7f10b806e9d0 2024-09-18T11:58:16.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b8098330 con 0x7f10b80a1230 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b809b960 con 0x7f10b8135300 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10bd7ce700 10 monclient(hunting): authenticate will time out at 1668.702448s 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10b77fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f10b8135300 0x7f10b8148150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:47900/0 (socket says 172.21.15.112:47900) 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10b77fe700 1 -- 172.21.15.112:0/561213794 learned_addr learned my addr 172.21.15.112:0/561213794 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10b67fc700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f10b80a1230 legacy=0x7f10b8144a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.857+0000 7f10b6ffd700 1 --1- 172.21.15.112:0/561213794 >> v1:172.21.15.112:6789/0 conn(0x7f10b806e9d0 0x7f10b813fb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35196/0 (socket says 172.21.15.112:35196) 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10b77fe700 1 -- 172.21.15.112:0/561213794 >> v1:172.21.15.112:6790/0 conn(0x7f10b8135300 legacy=0x7f10b8148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 client.?.objecter ms_handle_connect 0x7f10b8135300 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 client.?.objecter ms_handle_connect 0x7f10b806e9d0 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10b6ffd700 1 -- 172.21.15.112:0/561213794 >> v1:172.21.15.112:6789/0 conn(0x7f10b806e9d0 legacy=0x7f10b813fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:16.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010375044 0 0) 0x7f10b812f1e0 con 0x7f10b806e9d0 2024-09-18T11:58:16.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:16.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:16.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient(hunting): my global_id is 134172 2024-09-18T11:58:16.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a4004f10 con 0x7f10b806e9d0 2024-09-18T11:58:16.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3243999759 0 0) 0x7f10a4004f10 con 0x7f10b806e9d0 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10b812f1e0 con 0x7f10b806e9d0 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f10a8004830 con 0x7f10b806e9d0 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient(hunting): dump: 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2302679889 0 0) 0x7f10b812f1e0 con 0x7f10b806e9d0 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 >> v1:172.21.15.112:6790/0 conn(0x7f10b8135300 legacy=0x7f10b8148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 >> v1:172.21.15.129:6789/0 conn(0x7f10b80a1230 legacy=0x7f10b8144a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient: _finish_hunting 0 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 monclient: found mon.a 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b809a560 con 0x7f10b806e9d0 2024-09-18T11:58:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient: _finish_auth 0 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 5 monclient: authenticate success, global_id 134172 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 10 monclient: _renew_subs 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b8098aa0 con 0x7f10b806e9d0 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 10 client.134172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 10 monclient: _renew_subs 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b8099860 con 0x7f10b806e9d0 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 1 librados: init done 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10a8004e20 con 0x7f10b806e9d0 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10b4ff9700 4 set_mon_vals no callback set 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b809ac40 con 0x7f10b806e9d0 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f10a8005130 con 0x7f10b806e9d0 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 10 monclient: dump: 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.861+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f10a800f9b0 con 0x7f10b806e9d0 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.865+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(537..537 src has 1..537) v4 ==== 5750+0+0 (unknown 2527072992 0 0) 0x7f10a80440c0 con 0x7f10b806e9d0 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.865+0000 7f109effd700 10 client.134172.objecter ms_dispatch 0x7f10b8000fd0 osd_map(537..537 src has 1..537) v4 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.865+0000 7f109effd700 3 client.134172.objecter handle_osd_map got epochs [537,537] > 0 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.865+0000 7f109effd700 3 client.134172.objecter handle_osd_map decoding full epoch 537 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.865+0000 7f109effd700 20 client.134172.objecter dump_active .. 0 homeless 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.865+0000 7f10b67fc700 10 monclient: get_auth_request con 0x7f10a40380d0 auth_method 0 2024-09-18T11:58:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.865+0000 7f109effd700 10 client.134172.objecter ms_handle_connect 0x7f10a40380d0 2024-09-18T11:58:16.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.869+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f10a8014b90 con 0x7f10b806e9d0 2024-09-18T11:58:16.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.869+0000 7f109effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:16.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:16.869+0000 7f109effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:58:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:58:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f10b805f2e0 con 0x7f10b806e9d0 2024-09-18T11:58:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f109effd700 1 -- 172.21.15.112:0/561213794 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2760016987) 0x7f10a8014590 con 0x7f10b806e9d0 2024-09-18T11:58:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f109effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f109effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:17.101 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:58:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 10 librados: watch_flush enter 2024-09-18T11:58:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 10 librados: watch_flush exit 2024-09-18T11:58:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 20 client.134172.objecter shutdown clearing up homeless session... 2024-09-18T11:58:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 10 client.134172.objecter successfully canceled tick 2024-09-18T11:58:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 >> v1:172.21.15.129:6812/20806 conn(0x7f10a40380d0 legacy=0x7f10a403a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 10 monclient: shutdown 2024-09-18T11:58:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 >> v1:172.21.15.112:6789/0 conn(0x7f10b806e9d0 legacy=0x7f10b813fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 shutdown_connections 2024-09-18T11:58:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 >> 172.21.15.112:0/561213794 conn(0x7f10b80680d0 msgr2=0x7f10b81363c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.097+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 shutdown_connections 2024-09-18T11:58:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.101+0000 7f10bd7ce700 1 -- 172.21.15.112:0/561213794 wait complete. 2024-09-18T11:58:17.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:17.101+0000 7f10bd7ce700 1 librados: shutdown 2024-09-18T11:58:20.115 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:58:20.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.221+0000 7f14d93dc700 10 monclient: get_monmap_and_config 2024-09-18T11:58:20.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.221+0000 7f14d93dc700 10 monclient: build_initial_monmap 2024-09-18T11:58:20.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.221+0000 7f14d93dc700 10 monclient: monmap: 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:20.227338+0000 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:20.227338+0000 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:20.227 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.221+0000 7f14d93dc700 1 Processor -- start 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.221+0000 7f14d93dc700 1 -- start start 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.221+0000 7f14d93dc700 10 monclient: init 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 10 monclient(hunting): picked mon.noname-a con 0x7f14d40a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 10 monclient(hunting): picked mon.noname-c con 0x7f14d4096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 10 monclient(hunting): picked mon.noname-b con 0x7f14d413ca20 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:20.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d406e910 con 0x7f14d40a1ea0 2024-09-18T11:58:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d406e630 con 0x7f14d4096a60 2024-09-18T11:58:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d40a2270 con 0x7f14d413ca20 2024-09-18T11:58:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d93dc700 10 monclient(hunting): authenticate will time out at 1672.068359s 2024-09-18T11:58:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d37fe700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14d406c280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d3fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f14d40a1ea0 0x7f14d406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:57592/0 (socket says 172.21.15.112:57592) 2024-09-18T11:58:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d3fff700 1 -- 172.21.15.112:0/2739062141 learned_addr learned my addr 172.21.15.112:0/2739062141 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:20.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d8bdb700 1 --1- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6790/0 conn(0x7f14d413ca20 0x7f14d413ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50632/0 (socket says 172.21.15.112:50632) 2024-09-18T11:58:20.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d3fff700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6789/0 conn(0x7f14d40a1ea0 legacy=0x7f14d406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:20.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.225+0000 7f14d8bdb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6790/0 conn(0x7f14d413ca20 legacy=0x7f14d413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:20.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.429+0000 7f14d37fe700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14d406c280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:20.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:20.833+0000 7f14d37fe700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14d406c280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:21.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:21.637+0000 7f14d37fe700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14d406c280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:23.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 10 monclient(hunting): tick 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 1 monclient(hunting): continuing hunt 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6790/0 conn(0x7f14d413ca20 legacy=0x7f14d413ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14d406c280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6789/0 conn(0x7f14d40a1ea0 legacy=0x7f14d406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f14b4003810 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f14b4007490 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f14d4096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d406eed0 con 0x7f14b4007490 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d406e630 con 0x7f14d4096a60 2024-09-18T11:58:23.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b4010aa0 con 0x7f14b4003810 2024-09-18T11:58:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d1ffb700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d8bdb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d37fe700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6789/0 conn(0x7f14b4007490 legacy=0x7f14b4009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:23.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.225+0000 7f14d3fff700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6790/0 conn(0x7f14b4003810 legacy=0x7f14b4005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:23.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.429+0000 7f14d8bdb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:23.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:23.833+0000 7f14d8bdb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:24.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:24.637+0000 7f14d8bdb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:26.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 10 monclient(hunting): tick 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 1 monclient(hunting): continuing hunt 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6790/0 conn(0x7f14b4003810 legacy=0x7f14b4005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6789/0 conn(0x7f14b4007490 legacy=0x7f14b4009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f14b4010de0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f14d4096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f14b4003810 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b4011ce0 con 0x7f14b4010de0 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d406e630 con 0x7f14d4096a60 2024-09-18T11:58:26.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d37fe700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:26.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 1 -- 172.21.15.112:0/2739062141 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b4008910 con 0x7f14b4003810 2024-09-18T11:58:26.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d1ffb700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:26.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d3fff700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6789/0 conn(0x7f14b4010de0 legacy=0x7f14b40111b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:26.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.225+0000 7f14d8bdb700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6790/0 conn(0x7f14b4003810 legacy=0x7f14b4005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:26.429 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.429+0000 7f14d37fe700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:26.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:26.833+0000 7f14d37fe700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:27.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.637+0000 7f14d37fe700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:27.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487608332 0 0) 0x7f14b4008910 con 0x7f14b4003810 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient(hunting): my global_id is 104477 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c0003570 con 0x7f14b4003810 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 345239669 0 0) 0x7f14c0003570 con 0x7f14b4003810 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.129:6789/0 conn(0x7f14d4096a60 legacy=0x7f14b400d240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6789/0 conn(0x7f14b4010de0 legacy=0x7f14b40111b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: _finish_hunting 0 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 monclient: found mon.noname-b 2024-09-18T11:58:27.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b4009ab0 con 0x7f14b4003810 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: _finish_auth 0 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f14cc0019a0 con 0x7f14b4003810 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d93dc700 5 monclient: authenticate success, global_id 104477 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d93dc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: dump: 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:27.976 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d93dc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14cc001b00 con 0x7f14b4003810 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 1 -- 172.21.15.112:0/2739062141 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f14cc004830 con 0x7f14b4003810 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:27.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d2ffd700 10 monclient: dump: 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d93dc700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:27.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.969+0000 7f14d93dc700 4 set_mon_vals no callback set 2024-09-18T11:58:27.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient: shutdown 2024-09-18T11:58:27.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:27.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- 172.21.15.112:0/2739062141 >> v1:172.21.15.112:6790/0 conn(0x7f14b4003810 legacy=0x7f14b4005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:27.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- 172.21.15.112:0/2739062141 shutdown_connections 2024-09-18T11:58:27.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- 172.21.15.112:0/2739062141 >> 172.21.15.112:0/2739062141 conn(0x7f14d4067da0 msgr2=0x7f14d40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:27.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- 172.21.15.112:0/2739062141 shutdown_connections 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- 172.21.15.112:0/2739062141 wait complete. 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient: build_initial_monmap 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient: monmap: 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:27.981 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 librados: starting msgr at 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 librados: starting objecter 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 Processor -- start 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- start start 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 librados: setting wanted keys 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 librados: calling monclient init 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient: init 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient(hunting): picked mon.noname-c con 0x7f14b4003810 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient(hunting): picked mon.noname-b con 0x7f14b4010de0 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient(hunting): picked mon.noname-a con 0x7f14d4096a60 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d406e110 con 0x7f14d4096a60 2024-09-18T11:58:27.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d409dde0 con 0x7f14b4010de0 2024-09-18T11:58:27.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d4127ae0 con 0x7f14b4003810 2024-09-18T11:58:27.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:27.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.973+0000 7f14d93dc700 10 monclient(hunting): authenticate will time out at 1679.819858s 2024-09-18T11:58:27.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d37fe700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f14b4010de0 legacy=0x7f14d4146590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:27.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d8bdb700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f14d4096a60 0x7f14d414e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:57616/0 (socket says 172.21.15.112:57616) 2024-09-18T11:58:27.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d8bdb700 1 -- 172.21.15.112:0/538726311 learned_addr learned my addr 172.21.15.112:0/538726311 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d8bdb700 1 -- 172.21.15.112:0/538726311 >> v1:172.21.15.112:6789/0 conn(0x7f14d4096a60 legacy=0x7f14d414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 client.?.objecter ms_handle_connect 0x7f14d4096a60 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 client.?.objecter ms_handle_connect 0x7f14b4003810 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d3fff700 1 -- 172.21.15.112:0/538726311 >> v1:172.21.15.112:6790/0 conn(0x7f14b4003810 legacy=0x7f14d4145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911933511 0 0) 0x7f14d4127ae0 con 0x7f14b4003810 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:27.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient(hunting): my global_id is 104480 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14b4009c40 con 0x7f14b4003810 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 546381527 0 0) 0x7f14b4009c40 con 0x7f14b4003810 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d4127ae0 con 0x7f14b4003810 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f14bc0019f0 con 0x7f14b4003810 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient(hunting): dump: 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:27.987 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 551438261 0 0) 0x7f14d4127ae0 con 0x7f14b4003810 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 >> v1:172.21.15.129:6789/0 conn(0x7f14b4010de0 legacy=0x7f14d4146590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 >> v1:172.21.15.112:6789/0 conn(0x7f14d4096a60 legacy=0x7f14d414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient: _finish_hunting 0 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 monclient: found mon.c 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:27.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d4146ca0 con 0x7f14b4003810 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient: _finish_auth 0 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 5 monclient: authenticate success, global_id 104480 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 10 monclient: _renew_subs 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d4125230 con 0x7f14b4003810 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 10 client.104480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 10 monclient: _renew_subs 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d41258f0 con 0x7f14b4003810 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d93dc700 1 librados: init done 2024-09-18T11:58:27.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14bc001c90 con 0x7f14b4003810 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f14bc004d80 con 0x7f14b4003810 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14d1ffb700 4 set_mon_vals no callback set 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.977+0000 7f14cbfff700 10 monclient: dump: 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:27.990 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f14bc004ee0 con 0x7f14b4003810 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14d93dc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14d93dc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14d93dc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b80050b0 con 0x7f14b4003810 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(538..538 src has 1..538) v4 ==== 5750+0+0 (unknown 3886423803 0 0) 0x7f14bc043d80 con 0x7f14b4003810 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14cbfff700 10 client.104480.objecter ms_dispatch 0x7f14d4000fd0 osd_map(538..538 src has 1..538) v4 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14cbfff700 3 client.104480.objecter handle_osd_map got epochs [538,538] > 0 2024-09-18T11:58:27.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14cbfff700 3 client.104480.objecter handle_osd_map decoding full epoch 538 2024-09-18T11:58:27.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14cbfff700 20 client.104480.objecter dump_active .. 0 homeless 2024-09-18T11:58:27.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.981+0000 7f14d37fe700 10 monclient: get_auth_request con 0x7f14b4039670 auth_method 0 2024-09-18T11:58:27.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.985+0000 7f14cbfff700 10 client.104480.objecter ms_handle_connect 0x7f14b4039670 2024-09-18T11:58:27.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.985+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f14bc005600 con 0x7f14b4003810 2024-09-18T11:58:27.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.985+0000 7f14cbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:27.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:27.985+0000 7f14cbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.209+0000 7f14d93dc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:58:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.209+0000 7f14d93dc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.209+0000 7f14d93dc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.209+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f14b80058e0 con 0x7f14b4003810 2024-09-18T11:58:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.209+0000 7f14cbfff700 1 -- 172.21.15.112:0/538726311 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1108554950) 0x7f14bc014190 con 0x7f14b4003810 2024-09-18T11:58:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.209+0000 7f14cbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:58:28.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.209+0000 7f14cbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:28.215 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":64,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:58:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 10 librados: watch_flush enter 2024-09-18T11:58:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 10 librados: watch_flush exit 2024-09-18T11:58:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 20 client.104480.objecter shutdown clearing up homeless session... 2024-09-18T11:58:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 10 client.104480.objecter successfully canceled tick 2024-09-18T11:58:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 >> v1:172.21.15.129:6812/20806 conn(0x7f14b4039670 legacy=0x7f14b403bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 10 monclient: shutdown 2024-09-18T11:58:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 >> v1:172.21.15.112:6790/0 conn(0x7f14b4003810 legacy=0x7f14d4145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 shutdown_connections 2024-09-18T11:58:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 >> 172.21.15.112:0/538726311 conn(0x7f14d4067da0 msgr2=0x7f14d409fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 shutdown_connections 2024-09-18T11:58:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 1 -- 172.21.15.112:0/538726311 wait complete. 2024-09-18T11:58:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.213+0000 7f14d93dc700 1 librados: shutdown 2024-09-18T11:58:28.228 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T11:58:28.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T11:58:28.229 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.325+0000 7f5fe8f71700 10 monclient: get_monmap_and_config 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.325+0000 7f5fe8f71700 10 monclient: build_initial_monmap 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.325+0000 7f5fe8f71700 10 monclient: monmap: 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:28.331823+0000 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:28.331823+0000 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:28.331 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:28.332 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:28.332 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 1 Processor -- start 2024-09-18T11:58:28.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 1 -- start start 2024-09-18T11:58:28.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 10 monclient: init 2024-09-18T11:58:28.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:28.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:58:28.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 10 monclient(hunting): picked mon.noname-b con 0x7f5fe40a0380 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:28.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 10 monclient(hunting): picked mon.noname-a con 0x7f5fe40a0750 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 10 monclient(hunting): picked mon.noname-c con 0x7f5fe41385b0 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe4097270 con 0x7f5fe40a0750 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe40a3170 con 0x7f5fe41385b0 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe4096cb0 con 0x7f5fe40a0380 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe8f71700 10 monclient(hunting): authenticate will time out at 1680.173134s 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe2ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5fe40a0750 0x7f5fe40a0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:57624/0 (socket says 172.21.15.112:57624) 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe2ffd700 1 -- 172.21.15.112:0/287935757 learned_addr learned my addr 172.21.15.112:0/287935757 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:28.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe3fff700 1 -- 172.21.15.112:0/287935757 >> v1:172.21.15.129:6789/0 conn(0x7f5fe41385b0 legacy=0x7f5fe413a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:28.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe2ffd700 1 -- 172.21.15.112:0/287935757 >> v1:172.21.15.112:6789/0 conn(0x7f5fe40a0750 legacy=0x7f5fe40a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe37fe700 1 -- 172.21.15.112:0/287935757 >> v1:172.21.15.112:6790/0 conn(0x7f5fe40a0380 legacy=0x7f5fe4097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807326702 0 0) 0x7f5fe4097270 con 0x7f5fe40a0750 2024-09-18T11:58:28.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.329+0000 7f5fe27fc700 10 monclient(hunting): my global_id is 134175 2024-09-18T11:58:28.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fcc003570 con 0x7f5fe40a0750 2024-09-18T11:58:28.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210934611 0 0) 0x7f5fe4096cb0 con 0x7f5fe40a0380 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient(hunting): my global_id is 104483 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe4097270 con 0x7f5fe40a0380 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552273309 0 0) 0x7f5fe4097270 con 0x7f5fe40a0380 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 >> v1:172.21.15.129:6789/0 conn(0x7f5fe41385b0 legacy=0x7f5fe413a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 >> v1:172.21.15.112:6789/0 conn(0x7f5fe40a0750 legacy=0x7f5fe40a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: _finish_hunting 0 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 monclient: found mon.noname-b 2024-09-18T11:58:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40a1420 con 0x7f5fe40a0380 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: _finish_auth 0 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fd4001a30 con 0x7f5fe40a0380 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: dump: 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.338 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 5 monclient: authenticate success, global_id 104483 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd4001c10 con 0x7f5fe40a0380 2024-09-18T11:58:28.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 1 -- 172.21.15.112:0/287935757 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fd40020a0 con 0x7f5fe40a0380 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 4 set_mon_vals no callback set 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe27fc700 10 monclient: dump: 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.340 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.341 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.341 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.341 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.341 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.341 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.341 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 10 monclient: shutdown 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 -- 172.21.15.112:0/287935757 >> v1:172.21.15.112:6790/0 conn(0x7f5fe40a0380 legacy=0x7f5fe4097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 -- 172.21.15.112:0/287935757 shutdown_connections 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 -- 172.21.15.112:0/287935757 >> 172.21.15.112:0/287935757 conn(0x7f5fe40a5990 msgr2=0x7f5fe40a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 -- 172.21.15.112:0/287935757 shutdown_connections 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 -- 172.21.15.112:0/287935757 wait complete. 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 10 monclient: build_initial_monmap 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 10 monclient: monmap: 2024-09-18T11:58:28.349 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 librados: starting msgr at 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 librados: starting objecter 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 Processor -- start 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 -- start start 2024-09-18T11:58:28.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 librados: setting wanted keys 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 1 librados: calling monclient init 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.333+0000 7f5fe8f71700 10 monclient: init 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 10 monclient(hunting): picked mon.noname-b con 0x7f5fe40a0380 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 10 monclient(hunting): picked mon.noname-c con 0x7f5fe40a0750 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 10 monclient(hunting): picked mon.noname-a con 0x7f5fe41385b0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe40a2010 con 0x7f5fe41385b0 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe413ba50 con 0x7f5fe40a0380 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe37fe700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f5fe40a0380 legacy=0x7f5fe411c4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe411e200 con 0x7f5fe40a0750 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe3fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5fe41385b0 0x7f5fe4149e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:57640/0 (socket says 172.21.15.112:57640) 2024-09-18T11:58:28.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe3fff700 1 -- 172.21.15.112:0/3616041356 learned_addr learned my addr 172.21.15.112:0/3616041356 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe8f71700 10 monclient(hunting): authenticate will time out at 1680.180588s 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe3fff700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6789/0 conn(0x7f5fe41385b0 legacy=0x7f5fe4149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 client.?.objecter ms_handle_connect 0x7f5fe41385b0 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 client.?.objecter ms_handle_connect 0x7f5fe40a0750 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fe2ffd700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6790/0 conn(0x7f5fe40a0750 legacy=0x7f5fe4146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035677656 0 0) 0x7f5fe40a2010 con 0x7f5fe41385b0 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): my global_id is 134178 2024-09-18T11:58:28.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd0004f10 con 0x7f5fe41385b0 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361304372 0 0) 0x7f5fe411e200 con 0x7f5fe40a0750 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): my global_id is 104486 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe40a2010 con 0x7f5fe40a0750 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1507771578 0 0) 0x7f5fe40a2010 con 0x7f5fe40a0750 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe411e200 con 0x7f5fe40a0750 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fd8001c10 con 0x7f5fe40a0750 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): dump: 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.353 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 488948744 0 0) 0x7f5fd0004f10 con 0x7f5fe41385b0 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe40a2010 con 0x7f5fe41385b0 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fdc0047d0 con 0x7f5fe41385b0 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 10 monclient(hunting): dump: 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3785455707 0 0) 0x7f5fe411e200 con 0x7f5fe40a0750 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.337+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.129:6789/0 conn(0x7f5fe40a0380 legacy=0x7f5fe411c4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:58:28.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6789/0 conn(0x7f5fe41385b0 legacy=0x7f5fe4149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 monclient: _finish_hunting 0 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 1 monclient: found mon.c 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe411e980 con 0x7f5fe40a0750 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 monclient: _finish_auth 0 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 5 monclient: authenticate success, global_id 104486 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 10 monclient: _renew_subs 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe4137eb0 con 0x7f5fe40a0750 2024-09-18T11:58:28.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 10 client.104486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 10 monclient: _renew_subs 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe411dc90 con 0x7f5fe40a0750 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 1 librados: init done 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd8004be0 con 0x7f5fe40a0750 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fd8004f30 con 0x7f5fe40a0750 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe1ffb700 4 set_mon_vals no callback set 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 monclient: dump: 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.357 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5fc4006590 con 0x7f5fc4002840 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5fd8005090 con 0x7f5fe40a0750 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(538..538 src has 1..538) v4 ==== 5750+0+0 (unknown 3886423803 0 0) 0x7f5fd8043fc0 con 0x7f5fe40a0750 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 client.104486.objecter ms_dispatch 0x7f5fe4000fd0 osd_map(538..538 src has 1..538) v4 2024-09-18T11:58:28.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 3 client.104486.objecter handle_osd_map got epochs [538,538] > 0 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 3 client.104486.objecter handle_osd_map decoding full epoch 538 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 20 client.104486.objecter dump_active .. 0 homeless 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe3fff700 10 monclient: get_auth_request con 0x7f5fd00381c0 auth_method 0 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 client.104486.objecter ms_handle_connect 0x7f5fc4002840 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fcb7fe700 10 client.104486.objecter resend_mon_ops 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.341+0000 7f5fe37fe700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6789/0 conn(0x7f5fc4002840 legacy=0x7f5fc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640466496 0 0) 0x7f5fc4006590 con 0x7f5fc4002840 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 monclient(hunting): my global_id is 134181 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe411e200 con 0x7f5fc4002840 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3974131523 0 0) 0x7f5fe411e200 con 0x7f5fc4002840 2024-09-18T11:58:28.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fc4006590 con 0x7f5fc4002840 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fd4001390 con 0x7f5fc4002840 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 monclient: dump: 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.360 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3886410407 0 0) 0x7f5fc4006590 con 0x7f5fc4002840 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fc400a140 con 0x7f5fc4002840 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 client.104486.objecter ms_handle_connect 0x7f5fd00381c0 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5fd40068c0 con 0x7f5fc4002840 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6789/0 conn(0x7f5fc4002840 legacy=0x7f5fc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.345+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6789/0 conn(0x7f5fc4002840 legacy=0x7f5fc4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.353+0000 7f5fe8f71700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.353+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5fc4007550 con 0x7f5fc4006900 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 client.104486.objecter ms_handle_connect 0x7f5fc4006900 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 client.104486.objecter resend_mon_ops 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe37fe700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6789/0 conn(0x7f5fc4006900 legacy=0x7f5fc4006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111409460 0 0) 0x7f5fc4007550 con 0x7f5fc4006900 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 monclient(hunting): my global_id is 134184 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc4006590 con 0x7f5fc4006900 2024-09-18T11:58:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1525923381 0 0) 0x7f5fc4006590 con 0x7f5fc4006900 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fc4007550 con 0x7f5fc4006900 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5fd4002410 con 0x7f5fc4006900 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 monclient: dump: 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.364 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 736067955 0 0) 0x7f5fc4007550 con 0x7f5fc4006900 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5fc40087b0 con 0x7f5fc4006900 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3822 (unknown 0 0 92144148) 0x7f5fd4006b70 con 0x7f5fc4006900 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6789/0 conn(0x7f5fc4006900 legacy=0x7f5fc4006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fcb7fe700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6789/0 conn(0x7f5fc4006900 legacy=0x7f5fc4006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 64, 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:58:28.366 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:28.367 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:28.368 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:58:28.369 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:28.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:58:28.373 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:58:28.374 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.375 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.376 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:58:28.377 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:28.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 10 librados: watch_flush enter 2024-09-18T11:58:28.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 10 librados: watch_flush exit 2024-09-18T11:58:28.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 20 client.104486.objecter shutdown clearing up homeless session... 2024-09-18T11:58:28.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 10 client.104486.objecter successfully canceled tick 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.129:6812/20806 conn(0x7f5fd00381c0 legacy=0x7f5fd003a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 10 monclient: shutdown 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 >> v1:172.21.15.112:6790/0 conn(0x7f5fe40a0750 legacy=0x7f5fe4146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe3fff700 1 -- 172.21.15.112:0/3616041356 reap_dead start 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe3fff700 1 -- 172.21.15.112:0/3616041356 reap_dead start 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 shutdown_connections 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.357+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 >> 172.21.15.112:0/3616041356 conn(0x7f5fe40a5990 msgr2=0x7f5fe413a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.361+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 shutdown_connections 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.361+0000 7f5fe8f71700 1 -- 172.21.15.112:0/3616041356 wait complete. 2024-09-18T11:58:28.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.361+0000 7f5fe8f71700 1 librados: shutdown 2024-09-18T11:58:28.380 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:58:28.380 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T11:58:28.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:58:28.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient: get_monmap_and_config 2024-09-18T11:58:28.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient: build_initial_monmap 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient: monmap: 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:58:28.494543+0000 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:58:28.494543+0000 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:58:28.494 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 1 Processor -- start 2024-09-18T11:58:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 1 -- start start 2024-09-18T11:58:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient: init 2024-09-18T11:58:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:58:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient(hunting): picked mon.noname-a con 0x7fcd380a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient(hunting): picked mon.noname-c con 0x7fcd38096a60 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient(hunting): picked mon.noname-b con 0x7fcd3813ca20 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3806e910 con 0x7fcd380a1ea0 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3806e630 con 0x7fcd38096a60 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd380a2270 con 0x7fcd3813ca20 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3f57f700 10 monclient(hunting): authenticate will time out at 1680.335562s 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3dd7c700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7fcd38096a60 legacy=0x7fcd3806c280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3e57d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcd380a1ea0 0x7fcd3806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:57666/0 (socket says 172.21.15.112:57666) 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3e57d700 1 -- 172.21.15.112:0/3932352044 learned_addr learned my addr 172.21.15.112:0/3932352044 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.489+0000 7fcd3ed7e700 1 --1- 172.21.15.112:0/3932352044 >> v1:172.21.15.112:6790/0 conn(0x7fcd3813ca20 0x7fcd3813ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50682/0 (socket says 172.21.15.112:50682) 2024-09-18T11:58:28.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3e57d700 1 -- 172.21.15.112:0/3932352044 >> v1:172.21.15.112:6789/0 conn(0x7fcd380a1ea0 legacy=0x7fcd3806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3ed7e700 1 -- 172.21.15.112:0/3932352044 >> v1:172.21.15.112:6790/0 conn(0x7fcd3813ca20 legacy=0x7fcd3813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172176997 0 0) 0x7fcd380a2270 con 0x7fcd3813ca20 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient(hunting): my global_id is 104489 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd24003570 con 0x7fcd3813ca20 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316750099 0 0) 0x7fcd3806e910 con 0x7fcd380a1ea0 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient(hunting): my global_id is 134187 2024-09-18T11:58:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd380a2270 con 0x7fcd380a1ea0 2024-09-18T11:58:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339185765 0 0) 0x7fcd24003570 con 0x7fcd3813ca20 2024-09-18T11:58:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 >> v1:172.21.15.129:6789/0 conn(0x7fcd38096a60 legacy=0x7fcd3806c280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:58:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 >> v1:172.21.15.112:6789/0 conn(0x7fcd380a1ea0 legacy=0x7fcd3806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: _finish_hunting 0 2024-09-18T11:58:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 monclient: found mon.noname-b 2024-09-18T11:58:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3806eed0 con 0x7fcd3813ca20 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: _finish_auth 0 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd30001a00 con 0x7fcd3813ca20 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: dump: 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.504 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 5 monclient: authenticate success, global_id 104489 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd30001be0 con 0x7fcd3813ca20 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 1 -- 172.21.15.112:0/3932352044 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd300049d0 con 0x7fcd3813ca20 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 10 monclient: get_monmap_and_config success 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 4 set_mon_vals no callback set 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3d57b700 10 monclient: dump: 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.506 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 10 monclient: shutdown 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 1 -- 172.21.15.112:0/3932352044 >> v1:172.21.15.112:6790/0 conn(0x7fcd3813ca20 legacy=0x7fcd3813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 1 -- 172.21.15.112:0/3932352044 shutdown_connections 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 1 -- 172.21.15.112:0/3932352044 >> 172.21.15.112:0/3932352044 conn(0x7fcd38067da0 msgr2=0x7fcd380a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 1 -- 172.21.15.112:0/3932352044 shutdown_connections 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.493+0000 7fcd3f57f700 1 -- 172.21.15.112:0/3932352044 wait complete. 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient: build_initial_monmap 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient: monmap: 2024-09-18T11:58:28.507 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 librados: starting msgr at 2024-09-18T11:58:28.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 librados: starting objecter 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 Processor -- start 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 -- start start 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 librados: setting wanted keys 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 librados: calling monclient init 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient: init 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient: _reopen_session rank -1 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient(hunting): picked mon.noname-c con 0x7fcd38096a60 addr v1:172.21.15.112:6790/0 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient(hunting): picked mon.noname-a con 0x7fcd380a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient(hunting): picked mon.noname-b con 0x7fcd3813ca20 addr v1:172.21.15.129:6789/0 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3806e110 con 0x7fcd380a1ea0 2024-09-18T11:58:28.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3809db10 con 0x7fcd3813ca20 2024-09-18T11:58:28.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd381266f0 con 0x7fcd38096a60 2024-09-18T11:58:28.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3dd7c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcd380a1ea0 0x7fcd381239e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:57674/0 (socket says 172.21.15.112:57674) 2024-09-18T11:58:28.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient(hunting): _renew_subs 2024-09-18T11:58:28.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3dd7c700 1 -- 172.21.15.112:0/265544110 learned_addr learned my addr 172.21.15.112:0/265544110 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:58:28.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3e57d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fcd38096a60 0x7fcd381232d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50694/0 (socket says 172.21.15.112:50694) 2024-09-18T11:58:28.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3f57f700 10 monclient(hunting): authenticate will time out at 1680.342120s 2024-09-18T11:58:28.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.497+0000 7fcd3ed7e700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.129:6789/0 conn(0x7fcd3813ca20 legacy=0x7fcd3814ded0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:58:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3dd7c700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6789/0 conn(0x7fcd380a1ea0 legacy=0x7fcd381239e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 client.?.objecter ms_handle_connect 0x7fcd380a1ea0 2024-09-18T11:58:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3e57d700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6790/0 conn(0x7fcd38096a60 legacy=0x7fcd381232d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 client.?.objecter ms_handle_connect 0x7fcd38096a60 2024-09-18T11:58:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:58:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492918224 0 0) 0x7fcd3806e110 con 0x7fcd380a1ea0 2024-09-18T11:58:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): my global_id is 134190 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd1c004f10 con 0x7fcd380a1ea0 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700541673 0 0) 0x7fcd381266f0 con 0x7fcd38096a60 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): my global_id is 104492 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd3806e110 con 0x7fcd38096a60 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1725651402 0 0) 0x7fcd1c004f10 con 0x7fcd380a1ea0 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd381266f0 con 0x7fcd380a1ea0 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd34004860 con 0x7fcd380a1ea0 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): dump: 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:58:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2327532577 0 0) 0x7fcd3806e110 con 0x7fcd38096a60 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd1c004f10 con 0x7fcd38096a60 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd28004830 con 0x7fcd38096a60 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient(hunting): dump: 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3399194083 0 0) 0x7fcd1c004f10 con 0x7fcd38096a60 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.129:6789/0 conn(0x7fcd3813ca20 legacy=0x7fcd3814ded0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6789/0 conn(0x7fcd380a1ea0 legacy=0x7fcd381239e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 monclient: found mon.c 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd38126cb0 con 0x7fcd38096a60 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient: _finish_auth 0 2024-09-18T11:58:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:58:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 5 monclient: authenticate success, global_id 104492 2024-09-18T11:58:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 10 monclient: _renew_subs 2024-09-18T11:58:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3813c840 con 0x7fcd38096a60 2024-09-18T11:58:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 10 client.104492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 10 monclient: _renew_subs 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd381259a0 con 0x7fcd38096a60 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 1 librados: init done 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd280018f0 con 0x7fcd38096a60 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd28004de0 con 0x7fcd38096a60 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3cd7a700 4 set_mon_vals no callback set 2024-09-18T11:58:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 monclient: dump: 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcd20006590 con 0x7fcd20002840 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fcd2800f660 con 0x7fcd38096a60 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(538..538 src has 1..538) v4 ==== 5750+0+0 (unknown 3886423803 0 0) 0x7fcd28043e60 con 0x7fcd38096a60 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 10 client.104492.objecter ms_dispatch 0x7fcd38000fd0 osd_map(538..538 src has 1..538) v4 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 3 client.104492.objecter handle_osd_map got epochs [538,538] > 0 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 3 client.104492.objecter handle_osd_map decoding full epoch 538 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd2e7fc700 20 client.104492.objecter dump_active .. 0 homeless 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.501+0000 7fcd3ed7e700 10 monclient: get_auth_request con 0x7fcd1c038300 auth_method 0 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd3dd7c700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6790/0 conn(0x7fcd20002840 legacy=0x7fcd20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 client.104492.objecter ms_handle_connect 0x7fcd20002840 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 client.104492.objecter resend_mon_ops 2024-09-18T11:58:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879968887 0 0) 0x7fcd20006590 con 0x7fcd20002840 2024-09-18T11:58:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 monclient(hunting): my global_id is 104495 2024-09-18T11:58:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd1c004f10 con 0x7fcd20002840 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3059173671 0 0) 0x7fcd1c004f10 con 0x7fcd20002840 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd20006590 con 0x7fcd20002840 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd34004f00 con 0x7fcd20002840 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 monclient: dump: 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.525 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3141448537 0 0) 0x7fcd20006590 con 0x7fcd20002840 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd2000a140 con 0x7fcd20002840 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 client.104492.objecter ms_handle_connect 0x7fcd1c038300 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fcd34005250 con 0x7fcd20002840 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6790/0 conn(0x7fcd20002840 legacy=0x7fcd20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.505+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6790/0 conn(0x7fcd20002840 legacy=0x7fcd20004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:28.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd3f57f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:58:28.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcd20007550 con 0x7fcd20006900 2024-09-18T11:58:28.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 10 client.104492.objecter ms_handle_connect 0x7fcd20006900 2024-09-18T11:58:28.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 10 client.104492.objecter resend_mon_ops 2024-09-18T11:58:28.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd3dd7c700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6790/0 conn(0x7fcd20006900 legacy=0x7fcd20006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:58:28.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571018730 0 0) 0x7fcd20007550 con 0x7fcd20006900 2024-09-18T11:58:28.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 10 monclient(hunting): my global_id is 104498 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd20006590 con 0x7fcd20006900 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3512866077 0 0) 0x7fcd20006590 con 0x7fcd20006900 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd20007550 con 0x7fcd20006900 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.517+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcd34005910 con 0x7fcd20006900 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 10 monclient: dump: 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:58:28.529 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 880510500 0 0) 0x7fcd20007550 con 0x7fcd20006900 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcd200087b0 con 0x7fcd20006900 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4160 (unknown 0 0 832792718) 0x7fcd34007050 con 0x7fcd20006900 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6790/0 conn(0x7fcd20006900 legacy=0x7fcd20006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd2e7fc700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6790/0 conn(0x7fcd20006900 legacy=0x7fcd20006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 64, 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:58:28.532 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:58:28.533 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:58:28.534 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:58:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:28.537 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:58:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:58:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:58:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 6 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:58:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:58:28.542 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 10 librados: watch_flush enter 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 10 librados: watch_flush exit 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 20 client.104492.objecter shutdown clearing up homeless session... 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 10 client.104492.objecter successfully canceled tick 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.129:6812/20806 conn(0x7fcd1c038300 legacy=0x7fcd1c03a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 10 monclient: shutdown 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 >> v1:172.21.15.112:6790/0 conn(0x7fcd38096a60 legacy=0x7fcd381232d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3ed7e700 1 -- 172.21.15.112:0/265544110 reap_dead start 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3ed7e700 1 -- 172.21.15.112:0/265544110 reap_dead start 2024-09-18T11:58:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 shutdown_connections 2024-09-18T11:58:28.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 >> 172.21.15.112:0/265544110 conn(0x7fcd38067da0 msgr2=0x7fcd3806c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:58:28.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 shutdown_connections 2024-09-18T11:58:28.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 1 -- 172.21.15.112:0/265544110 wait complete. 2024-09-18T11:58:28.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:58:28.521+0000 7fcd3f57f700 1 librados: shutdown 2024-09-18T11:58:28.545 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T11:58:28.545 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T11:59:09.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:59:09.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9002 ms) 2024-09-18T11:59:09.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-18T11:59:09.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:59:09.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6151 ms) 2024-09-18T11:59:09.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-18T11:59:09.402 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (4866 ms) 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25258 ms) 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13262 ms) 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (19569 ms) 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-18T11:59:09.403 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000012 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (25945 ms) 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = quincy 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5982 ms) 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (670447 ms total) 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1467 ms) 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10966 ms) 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-18T11:59:09.404 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17531 ms) 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (29966 ms total) 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6944 ms) 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8764 ms) 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrub... 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (25361 ms) 2024-09-18T11:59:09.405 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-18T11:59:09.406 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (24523 ms) 2024-09-18T11:59:09.406 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-18T11:59:09.406 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8152 ms) 2024-09-18T11:59:09.406 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-18T11:59:09.406 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7870 ms) 2024-09-18T11:59:09.406 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-18T11:59:09.406 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10067 ms) 2024-09-18T11:59:09.406 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-18T11:59:58.546 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T11:59:58.546 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T11:59:58.546 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T11:59:58.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T11:59:58.550 INFO:tasks.ceph.mon.b:Started 2024-09-18T11:59:58.551 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T11:59:58.551 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T11:59:58.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient: get_monmap_and_config 2024-09-18T11:59:58.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient: build_initial_monmap 2024-09-18T11:59:58.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient: monmap: 2024-09-18T11:59:58.660 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:58.660 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:58.661 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:59:58.660661+0000 2024-09-18T11:59:58.661 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:59:58.660661+0000 2024-09-18T11:59:58.661 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:58.661 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:58.661 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:58.661 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:59:58.662 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:59:58.662 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:58.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 1 Processor -- start 2024-09-18T11:59:58.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 1 -- start start 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient: init 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient(hunting): picked mon.noname-c con 0x7f5ae406aa40 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient(hunting): picked mon.noname-b con 0x7f5ae40a1e80 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient(hunting): picked mon.noname-a con 0x7f5ae41386e0 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae4097270 con 0x7f5ae41386e0 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae406ae10 con 0x7f5ae406aa40 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae4097550 con 0x7f5ae40a1e80 2024-09-18T11:59:58.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aec645700 10 monclient(hunting): authenticate will time out at 1770.502174s 2024-09-18T11:59:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aea3e1700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f5ae406aa40 legacy=0x7f5ae40a0a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:59:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aeabe2700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5ae41386e0 0x7f5ae413aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60490/0 (socket says 172.21.15.112:60490) 2024-09-18T11:59:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5aeabe2700 1 -- 172.21.15.112:0/3795218909 learned_addr learned my addr 172.21.15.112:0/3795218909 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.659+0000 7f5ae9be0700 1 --1- 172.21.15.112:0/3795218909 >> v1:172.21.15.112:6790/0 conn(0x7f5ae40a1e80 0x7f5ae40a1160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45742/0 (socket says 172.21.15.112:45742) 2024-09-18T11:59:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aeabe2700 1 -- 172.21.15.112:0/3795218909 >> v1:172.21.15.112:6789/0 conn(0x7f5ae41386e0 legacy=0x7f5ae413aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae9be0700 1 -- 172.21.15.112:0/3795218909 >> v1:172.21.15.112:6790/0 conn(0x7f5ae40a1e80 legacy=0x7f5ae40a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671958795 0 0) 0x7f5ae4097270 con 0x7f5ae41386e0 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient(hunting): my global_id is 134193 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5acc003570 con 0x7f5ae41386e0 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260330583 0 0) 0x7f5ae4097550 con 0x7f5ae40a1e80 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient(hunting): my global_id is 104501 2024-09-18T11:59:58.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae4097270 con 0x7f5ae40a1e80 2024-09-18T11:59:58.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3660203880 0 0) 0x7f5acc003570 con 0x7f5ae41386e0 2024-09-18T11:59:58.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 >> v1:172.21.15.112:6790/0 conn(0x7f5ae40a1e80 legacy=0x7f5ae40a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:58.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 >> v1:172.21.15.129:6789/0 conn(0x7f5ae406aa40 legacy=0x7f5ae40a0a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T11:59:58.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: _finish_hunting 0 2024-09-18T11:59:58.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 monclient: found mon.noname-a 2024-09-18T11:59:58.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:58.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:59:58.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4096a60 con 0x7f5ae41386e0 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: _finish_auth 0 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 5 monclient: authenticate success, global_id 134193 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5adc001940 con 0x7f5ae41386e0 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: dump: 2024-09-18T11:59:58.674 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:58.675 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:58.675 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.675 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.675 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:58.675 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:58.675 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:58.675 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:58.675 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:58.676 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:58.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:58.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5adc001bd0 con 0x7f5ae41386e0 2024-09-18T11:59:58.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:58.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 10 monclient: get_monmap_and_config success 2024-09-18T11:59:58.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 4 set_mon_vals no callback set 2024-09-18T11:59:58.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 1 -- 172.21.15.112:0/3795218909 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5adc004a90 con 0x7f5ae41386e0 2024-09-18T11:59:58.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5ae93df700 10 monclient: dump: 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:58.677 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:58.678 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:58.678 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:58.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 10 monclient: shutdown 2024-09-18T11:59:58.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:58.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 1 -- 172.21.15.112:0/3795218909 >> v1:172.21.15.112:6789/0 conn(0x7f5ae41386e0 legacy=0x7f5ae413aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:58.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 1 -- 172.21.15.112:0/3795218909 shutdown_connections 2024-09-18T11:59:58.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 1 -- 172.21.15.112:0/3795218909 >> 172.21.15.112:0/3795218909 conn(0x7f5ae40a5d10 msgr2=0x7f5ae40a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:58.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 1 -- 172.21.15.112:0/3795218909 shutdown_connections 2024-09-18T11:59:58.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 1 -- 172.21.15.112:0/3795218909 wait complete. 2024-09-18T11:59:58.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 10 monclient: build_initial_monmap 2024-09-18T11:59:58.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 10 monclient: monmap: 2024-09-18T11:59:58.679 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:58.680 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:59:58.680 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:59:58.680 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:59:58.680 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:58.680 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:58.680 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:58.680 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:59:58.681 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:59:58.681 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:58.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 1 librados: starting msgr at 2024-09-18T11:59:58.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.663+0000 7f5aec645700 1 librados: starting objecter 2024-09-18T11:59:58.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 1 Processor -- start 2024-09-18T11:59:58.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 1 -- start start 2024-09-18T11:59:58.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 1 librados: setting wanted keys 2024-09-18T11:59:58.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 1 librados: calling monclient init 2024-09-18T11:59:58.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 10 monclient: init 2024-09-18T11:59:58.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:58.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:59:58.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 10 monclient(hunting): picked mon.noname-a con 0x7f5ae406aa40 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:58.682 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 10 monclient(hunting): picked mon.noname-b con 0x7f5ae40a1e80 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:58.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 10 monclient(hunting): picked mon.noname-c con 0x7f5ae41386e0 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:58.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae4134fa0 con 0x7f5ae406aa40 2024-09-18T11:59:58.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae41381b0 con 0x7f5ae40a1e80 2024-09-18T11:59:58.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae4134cc0 con 0x7f5ae41386e0 2024-09-18T11:59:58.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:58.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aec645700 10 monclient(hunting): authenticate will time out at 1770.509044s 2024-09-18T11:59:58.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ae9be0700 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f5ae40a1e80 legacy=0x7f5ae41469a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.129:6789/0 2024-09-18T11:59:58.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aea3e1700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5ae406aa40 0x7f5ae4137aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60502/0 (socket says 172.21.15.112:60502) 2024-09-18T11:59:58.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aea3e1700 1 -- 172.21.15.112:0/3568940106 learned_addr learned my addr 172.21.15.112:0/3568940106 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:58.685 INFO:tasks.ceph.mon.b.smithi129.stderr:2024-09-18T11:59:58.665+0000 7fa83f438a00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 570 -> 0 (sync_force? mon rebuild?) 2024-09-18T11:59:58.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aea3e1700 1 -- 172.21.15.112:0/3568940106 >> v1:172.21.15.112:6789/0 conn(0x7f5ae406aa40 legacy=0x7f5ae4137aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:58.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 client.?.objecter ms_handle_connect 0x7f5ae406aa40 2024-09-18T11:59:58.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5aeabe2700 1 -- 172.21.15.112:0/3568940106 >> v1:172.21.15.112:6790/0 conn(0x7f5ae41386e0 legacy=0x7f5ae414a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:58.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:58.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 client.?.objecter ms_handle_connect 0x7f5ae41386e0 2024-09-18T11:59:58.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:58.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728096845 0 0) 0x7f5ae4134fa0 con 0x7f5ae406aa40 2024-09-18T11:59:58.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 monclient(hunting): my global_id is 134196 2024-09-18T11:59:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad0004f10 con 0x7f5ae406aa40 2024-09-18T11:59:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900324326 0 0) 0x7f5ae4134cc0 con 0x7f5ae41386e0 2024-09-18T11:59:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 10 monclient(hunting): my global_id is 104504 2024-09-18T11:59:58.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.667+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae4134fa0 con 0x7f5ae41386e0 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2673572741 0 0) 0x7f5ad0004f10 con 0x7f5ae406aa40 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ae4134cc0 con 0x7f5ae406aa40 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5ad4004920 con 0x7f5ae406aa40 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient(hunting): dump: 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:58.697 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1630897948 0 0) 0x7f5ae4134fa0 con 0x7f5ae41386e0 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ad0004f10 con 0x7f5ae41386e0 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5adc004830 con 0x7f5ae41386e0 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:58.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient(hunting): dump: 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:58.699 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1874134958 0 0) 0x7f5ae4134cc0 con 0x7f5ae406aa40 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 >> v1:172.21.15.112:6790/0 conn(0x7f5ae41386e0 legacy=0x7f5ae414a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 >> v1:172.21.15.129:6789/0 conn(0x7f5ae40a1e80 legacy=0x7f5ae41469a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient: _finish_hunting 0 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 monclient: found mon.a 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:58.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4135280 con 0x7f5ae406aa40 2024-09-18T11:59:58.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient: _finish_auth 0 2024-09-18T11:59:58.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:58.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 5 monclient: authenticate success, global_id 134196 2024-09-18T11:59:58.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 10 monclient: _renew_subs 2024-09-18T11:59:58.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:58.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae41331f0 con 0x7f5ae406aa40 2024-09-18T11:59:58.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 10 client.134196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:59:58.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 10 monclient: _renew_subs 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae4133fc0 con 0x7f5ae406aa40 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 1 librados: init done 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ad40050a0 con 0x7f5ae406aa40 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5ad40053b0 con 0x7f5ae406aa40 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ae8bde700 4 set_mon_vals no callback set 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 monclient: dump: 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.702 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5ad400fc30 con 0x7f5ae406aa40 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac40050b0 con 0x7f5ae406aa40 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(624..624 src has 1..624) v4 ==== 5793+0+0 (unknown 1762967044 0 0) 0x7f5ad4044430 con 0x7f5ae406aa40 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 10 client.134196.objecter ms_dispatch 0x7f5ae4000fd0 osd_map(624..624 src has 1..624) v4 2024-09-18T11:59:58.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 3 client.134196.objecter handle_osd_map got epochs [624,624] > 0 2024-09-18T11:59:58.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 3 client.134196.objecter handle_osd_map decoding full epoch 624 2024-09-18T11:59:58.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ada7fc700 20 client.134196.objecter dump_active .. 0 homeless 2024-09-18T11:59:58.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.671+0000 7f5ae9be0700 10 monclient: get_auth_request con 0x7f5ad0038320 auth_method 0 2024-09-18T11:59:58.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.675+0000 7f5ada7fc700 10 client.134196.objecter ms_handle_connect 0x7f5ad0038320 2024-09-18T11:59:58.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.679+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f5ad40149f0 con 0x7f5ae406aa40 2024-09-18T11:59:58.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.679+0000 7f5ada7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:58.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.679+0000 7f5ada7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:59:58.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.915+0000 7f5aec645700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T11:59:58.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.915+0000 7f5aec645700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T11:59:58.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.915+0000 7f5aec645700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:58.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.915+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5ac40058e0 con 0x7f5ae406aa40 2024-09-18T11:59:58.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.915+0000 7f5ada7fc700 1 -- 172.21.15.112:0/3568940106 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 4253514435) 0x7f5ad4018fb0 con 0x7f5ae406aa40 2024-09-18T11:59:58.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.915+0000 7f5ada7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T11:59:58.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.915+0000 7f5ada7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:59:58.919 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T11:59:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 10 librados: watch_flush enter 2024-09-18T11:59:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 10 librados: watch_flush exit 2024-09-18T11:59:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 20 client.134196.objecter shutdown clearing up homeless session... 2024-09-18T11:59:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 10 client.134196.objecter successfully canceled tick 2024-09-18T11:59:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 >> v1:172.21.15.129:6812/20806 conn(0x7f5ad0038320 legacy=0x7f5ad003a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 10 monclient: shutdown 2024-09-18T11:59:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:58.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 >> v1:172.21.15.112:6789/0 conn(0x7f5ae406aa40 legacy=0x7f5ae4137aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:58.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 shutdown_connections 2024-09-18T11:59:58.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 >> 172.21.15.112:0/3568940106 conn(0x7f5ae40a5d10 msgr2=0x7f5ae413a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:58.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 shutdown_connections 2024-09-18T11:59:58.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 1 -- 172.21.15.112:0/3568940106 wait complete. 2024-09-18T11:59:58.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:58.919+0000 7f5aec645700 1 librados: shutdown 2024-09-18T11:59:58.932 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T11:59:58.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T11:59:58.932 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T11:59:59.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.043+0000 7f1a89cb6700 10 monclient: get_monmap_and_config 2024-09-18T11:59:59.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.043+0000 7f1a89cb6700 10 monclient: build_initial_monmap 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.043+0000 7f1a89cb6700 10 monclient: monmap: 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:59:59.045683+0000 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:59:59.045683+0000 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:59:59.045 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.043+0000 7f1a89cb6700 1 Processor -- start 2024-09-18T11:59:59.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.043+0000 7f1a89cb6700 1 -- start start 2024-09-18T11:59:59.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.043+0000 7f1a89cb6700 10 monclient: init 2024-09-18T11:59:59.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:59.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:59:59.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 10 monclient(hunting): picked mon.noname-b con 0x7f1a84067680 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:59.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 10 monclient(hunting): picked mon.noname-c con 0x7f1a840994f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:59.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 10 monclient(hunting): picked mon.noname-a con 0x7f1a84097880 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:59.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a84067a50 con 0x7f1a84097880 2024-09-18T11:59:59.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a84098b90 con 0x7f1a840994f0 2024-09-18T11:59:59.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8409dee0 con 0x7f1a84067680 2024-09-18T11:59:59.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:59.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a88cb4700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1a84067680 0x7f1a84096a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:45758/0 (socket says 172.21.15.112:45758) 2024-09-18T11:59:59.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a88cb4700 1 -- 172.21.15.112:0/4082794526 learned_addr learned my addr 172.21.15.112:0/4082794526 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:59.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 10 monclient(hunting): authenticate will time out at 1770.888472s 2024-09-18T11:59:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a894b5700 1 -- 172.21.15.112:0/4082794526 >> v1:172.21.15.112:6789/0 conn(0x7f1a84097880 legacy=0x7f1a84137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a88cb4700 1 -- 172.21.15.112:0/4082794526 >> v1:172.21.15.112:6790/0 conn(0x7f1a84067680 legacy=0x7f1a84096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218906702 0 0) 0x7f1a84067a50 con 0x7f1a84097880 2024-09-18T11:59:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient(hunting): my global_id is 134199 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a74003570 con 0x7f1a84097880 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a83fff700 1 -- 172.21.15.112:0/4082794526 >> v1:172.21.15.129:6789/0 conn(0x7f1a840994f0 legacy=0x7f1a84097170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632267744 0 0) 0x7f1a74003570 con 0x7f1a84097880 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 >> v1:172.21.15.112:6790/0 conn(0x7f1a84067680 legacy=0x7f1a84096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 >> v1:172.21.15.129:6789/0 conn(0x7f1a840994f0 legacy=0x7f1a84097170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient: _finish_hunting 0 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 1 monclient: found mon.noname-a 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8409e120 con 0x7f1a84097880 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient: _finish_auth 0 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:59.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 5 monclient: authenticate success, global_id 134199 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1a7c001dd0 con 0x7f1a84097880 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a837fe700 10 monclient: dump: 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.053 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.047+0000 7f1a89cb6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a7c001fb0 con 0x7f1a84097880 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 10 monclient: get_monmap_and_config success 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a837fe700 1 -- 172.21.15.112:0/4082794526 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1a7c002420 con 0x7f1a84097880 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 4 set_mon_vals no callback set 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a837fe700 10 monclient: dump: 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.054 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.055 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.055 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.055 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.055 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.055 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.055 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 10 monclient: shutdown 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 1 -- 172.21.15.112:0/4082794526 >> v1:172.21.15.112:6789/0 conn(0x7f1a84097880 legacy=0x7f1a84137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 1 -- 172.21.15.112:0/4082794526 shutdown_connections 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 1 -- 172.21.15.112:0/4082794526 >> 172.21.15.112:0/4082794526 conn(0x7f1a840a1e30 msgr2=0x7f1a840a2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 1 -- 172.21.15.112:0/4082794526 shutdown_connections 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 1 -- 172.21.15.112:0/4082794526 wait complete. 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 10 monclient: build_initial_monmap 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 10 monclient: monmap: 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:59.057 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 1 librados: starting msgr at 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.051+0000 7f1a89cb6700 1 librados: starting objecter 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 1 Processor -- start 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 1 -- start start 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 1 librados: setting wanted keys 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 1 librados: calling monclient init 2024-09-18T11:59:59.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 10 monclient: init 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 10 monclient(hunting): picked mon.noname-a con 0x7f1a84067680 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 10 monclient(hunting): picked mon.noname-c con 0x7f1a84097880 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 10 monclient(hunting): picked mon.noname-b con 0x7f1a840994f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a84097ca0 con 0x7f1a84067680 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a88cb4700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1a84067680 0x7f1a8411e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60512/0 (socket says 172.21.15.112:60512) 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a88cb4700 1 -- 172.21.15.112:0/3690009868 learned_addr learned my addr 172.21.15.112:0/3690009868 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8411eed0 con 0x7f1a840994f0 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8409b0b0 con 0x7f1a84097880 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:59.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a89cb6700 10 monclient(hunting): authenticate will time out at 1770.897729s 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a88cb4700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6789/0 conn(0x7f1a84067680 legacy=0x7f1a8411e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a80ff9700 10 client.?.objecter ms_handle_connect 0x7f1a84067680 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a80ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a80ff9700 10 client.?.objecter ms_handle_connect 0x7f1a84097880 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a80ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.055+0000 7f1a83fff700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6790/0 conn(0x7f1a84097880 legacy=0x7f1a8411e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719168315 0 0) 0x7f1a84097ca0 con 0x7f1a84067680 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): my global_id is 134202 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a68004f10 con 0x7f1a84067680 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401339484 0 0) 0x7f1a8409b0b0 con 0x7f1a84097880 2024-09-18T11:59:59.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): my global_id is 104507 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a84097ca0 con 0x7f1a84097880 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 client.?.objecter ms_handle_connect 0x7f1a840994f0 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a894b5700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.129:6789/0 conn(0x7f1a840994f0 legacy=0x7f1a84147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304337577 0 0) 0x7f1a68004f10 con 0x7f1a84067680 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a8409b0b0 con 0x7f1a84067680 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1a78001b60 con 0x7f1a84067680 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): dump: 2024-09-18T11:59:59.064 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4000006973 0 0) 0x7f1a84097ca0 con 0x7f1a84097880 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a68004f10 con 0x7f1a84097880 2024-09-18T11:59:59.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1a70001c00 con 0x7f1a84097880 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient(hunting): dump: 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 356134854 0 0) 0x7f1a8409b0b0 con 0x7f1a84067680 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6790/0 conn(0x7f1a84097880 legacy=0x7f1a8411e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.129:6789/0 conn(0x7f1a840994f0 legacy=0x7f1a84147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient: _finish_hunting 0 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 monclient: found mon.a 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8411f2e0 con 0x7f1a84067680 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient: _finish_auth 0 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 5 monclient: authenticate success, global_id 134202 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 10 monclient: _renew_subs 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a841206c0 con 0x7f1a84067680 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 10 client.134202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:59:59.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 10 monclient: _renew_subs 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8411f760 con 0x7f1a84067680 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 1 librados: init done 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1a84138b10 con 0x7f1a84138740 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a78004be0 con 0x7f1a84067680 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1a78004f30 con 0x7f1a84067680 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 monclient: dump: 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.068 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f1a78005140 con 0x7f1a84067680 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a82ffd700 4 set_mon_vals no callback set 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(624..624 src has 1..624) v4 ==== 5793+0+0 (unknown 1762967044 0 0) 0x7f1a780440f0 con 0x7f1a84067680 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 10 client.134202.objecter ms_dispatch 0x7f1a84000fd0 osd_map(624..624 src has 1..624) v4 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 3 client.134202.objecter handle_osd_map got epochs [624,624] > 0 2024-09-18T11:59:59.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 3 client.134202.objecter handle_osd_map decoding full epoch 624 2024-09-18T11:59:59.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.059+0000 7f1a80ff9700 20 client.134202.objecter dump_active .. 0 homeless 2024-09-18T11:59:59.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a894b5700 10 monclient: get_auth_request con 0x7f1a68038360 auth_method 0 2024-09-18T11:59:59.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a83fff700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6789/0 conn(0x7f1a84138740 legacy=0x7f1a84118490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 client.134202.objecter ms_handle_connect 0x7f1a84138740 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 client.134202.objecter resend_mon_ops 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351023474 0 0) 0x7f1a84138b10 con 0x7f1a84138740 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 monclient(hunting): my global_id is 134205 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a8409b0b0 con 0x7f1a84138740 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 786593209 0 0) 0x7f1a8409b0b0 con 0x7f1a84138740 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a84138b10 con 0x7f1a84138740 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1a700026c0 con 0x7f1a84138740 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 monclient: dump: 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 433854630 0 0) 0x7f1a84138b10 con 0x7f1a84138740 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a8405f2e0 con 0x7f1a84138740 2024-09-18T11:59:59.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f1a700028a0 con 0x7f1a84138740 2024-09-18T11:59:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:59:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6789/0 conn(0x7f1a84138740 legacy=0x7f1a84118490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6789/0 conn(0x7f1a84138740 legacy=0x7f1a84118490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:59:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.063+0000 7f1a80ff9700 10 client.134202.objecter ms_handle_connect 0x7f1a68038360 2024-09-18T11:59:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.071+0000 7f1a89cb6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:59:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.071+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1a840a0680 con 0x7f1a8402ad10 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 client.134202.objecter ms_handle_connect 0x7f1a8402ad10 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 client.134202.objecter resend_mon_ops 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a83fff700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6789/0 conn(0x7f1a8402ad10 legacy=0x7f1a84118ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246961257 0 0) 0x7f1a840a0680 con 0x7f1a8402ad10 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 monclient(hunting): my global_id is 134208 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a84138b10 con 0x7f1a8402ad10 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1114948666 0 0) 0x7f1a84138b10 con 0x7f1a8402ad10 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a840a0680 con 0x7f1a8402ad10 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1a700059b0 con 0x7f1a8402ad10 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 monclient: dump: 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3343629165 0 0) 0x7f1a840a0680 con 0x7f1a8402ad10 2024-09-18T11:59:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1a84064ec0 con 0x7f1a8402ad10 2024-09-18T11:59:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4003 (unknown 0 0 1549591954) 0x7f1a70005b10 con 0x7f1a8402ad10 2024-09-18T11:59:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:59:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:59:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6789/0 conn(0x7f1a8402ad10 legacy=0x7f1a84118ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a80ff9700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6789/0 conn(0x7f1a8402ad10 legacy=0x7f1a84118ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 70, 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:59:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.086 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:59:59.087 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.088 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:59:59.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.090 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:59:59.091 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:59:59.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 10 librados: watch_flush enter 2024-09-18T11:59:59.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 10 librados: watch_flush exit 2024-09-18T11:59:59.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 20 client.134202.objecter shutdown clearing up homeless session... 2024-09-18T11:59:59.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 10 client.134202.objecter successfully canceled tick 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.129:6812/20806 conn(0x7f1a68038360 legacy=0x7f1a6803a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 10 monclient: shutdown 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 >> v1:172.21.15.112:6789/0 conn(0x7f1a84067680 legacy=0x7f1a8411e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a894b5700 1 -- 172.21.15.112:0/3690009868 reap_dead start 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a894b5700 1 -- 172.21.15.112:0/3690009868 reap_dead start 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 shutdown_connections 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 >> 172.21.15.112:0/3690009868 conn(0x7f1a840a1e30 msgr2=0x7f1a8409b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 shutdown_connections 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 1 -- 172.21.15.112:0/3690009868 wait complete. 2024-09-18T11:59:59.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.075+0000 7f1a89cb6700 1 librados: shutdown 2024-09-18T11:59:59.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:59:59.096 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T11:59:59.096 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T11:59:59.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient: get_monmap_and_config 2024-09-18T11:59:59.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient: build_initial_monmap 2024-09-18T11:59:59.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient: monmap: 2024-09-18T11:59:59.201 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:59.201 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.201 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:59:59.201772+0000 2024-09-18T11:59:59.201 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:59:59.201772+0000 2024-09-18T11:59:59.201 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:59.201 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.202 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:59.202 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:59:59.202 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:59:59.202 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 1 Processor -- start 2024-09-18T11:59:59.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 1 -- start start 2024-09-18T11:59:59.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient: init 2024-09-18T11:59:59.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient(hunting): picked mon.noname-c con 0x7f96600a21c0 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient(hunting): picked mon.noname-a con 0x7f966009fdf0 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient(hunting): picked mon.noname-b con 0x7f96600a01f0 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966006b7c0 con 0x7f966009fdf0 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96600a2590 con 0x7f96600a21c0 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966009beb0 con 0x7f96600a01f0 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:59.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f966727c700 10 monclient(hunting): authenticate will time out at 1771.042915s 2024-09-18T11:59:59.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f9665819700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f96600a01f0 0x7f966013f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59482/0 (socket says 172.21.15.112:59482) 2024-09-18T11:59:59.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.199+0000 7f9665819700 1 -- 172.21.15.112:0/1988662592 learned_addr learned my addr 172.21.15.112:0/1988662592 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:59.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9665819700 1 -- 172.21.15.112:0/1988662592 >> v1:172.21.15.112:6790/0 conn(0x7f96600a01f0 legacy=0x7f966013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9664817700 1 -- 172.21.15.112:0/1988662592 >> v1:172.21.15.112:6789/0 conn(0x7f966009fdf0 legacy=0x7f966013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9665018700 1 -- 172.21.15.112:0/1988662592 >> v1:172.21.15.129:6789/0 conn(0x7f96600a21c0 legacy=0x7f966009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505121544 0 0) 0x7f966006b7c0 con 0x7f966009fdf0 2024-09-18T11:59:59.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient(hunting): my global_id is 134211 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f964c003570 con 0x7f966009fdf0 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385090600 0 0) 0x7f96600a2590 con 0x7f96600a21c0 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient(hunting): my global_id is 144100 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966006b7c0 con 0x7f96600a21c0 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 346950516 0 0) 0x7f964c003570 con 0x7f966009fdf0 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 >> v1:172.21.15.112:6790/0 conn(0x7f96600a01f0 legacy=0x7f966013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 >> v1:172.21.15.129:6789/0 conn(0x7f96600a21c0 legacy=0x7f966009f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: _finish_hunting 0 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 monclient: found mon.noname-a 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96600a0fb0 con 0x7f966009fdf0 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: _finish_auth 0 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f965c001a50 con 0x7f966009fdf0 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 5 monclient: authenticate success, global_id 134211 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:59:59.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: dump: 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f965c001c30 con 0x7f966009fdf0 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:59.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 10 monclient: get_monmap_and_config success 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 4 set_mon_vals no callback set 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 1 -- 172.21.15.112:0/1988662592 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f965c0020a0 con 0x7f966009fdf0 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f9653fff700 10 monclient: dump: 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.213 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.214 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.214 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 10 monclient: shutdown 2024-09-18T11:59:59.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:59.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 1 -- 172.21.15.112:0/1988662592 >> v1:172.21.15.112:6789/0 conn(0x7f966009fdf0 legacy=0x7f966013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 1 -- 172.21.15.112:0/1988662592 shutdown_connections 2024-09-18T11:59:59.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 1 -- 172.21.15.112:0/1988662592 >> 172.21.15.112:0/1988662592 conn(0x7f96600680d0 msgr2=0x7f96600a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:59.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 1 -- 172.21.15.112:0/1988662592 shutdown_connections 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.203+0000 7f966727c700 1 -- 172.21.15.112:0/1988662592 wait complete. 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient: build_initial_monmap 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient: monmap: 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:59.215 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 librados: starting msgr at 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 librados: starting objecter 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 Processor -- start 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 -- start start 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 librados: setting wanted keys 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 librados: calling monclient init 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient: init 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:59.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient(hunting): picked mon.noname-a con 0x7f966009fdf0 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient(hunting): picked mon.noname-b con 0x7f96600a01f0 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient(hunting): picked mon.noname-c con 0x7f96600a21c0 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966006edd0 con 0x7f966009fdf0 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966009d8d0 con 0x7f96600a01f0 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9660146a80 con 0x7f96600a21c0 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f966727c700 10 monclient(hunting): authenticate will time out at 1771.050063s 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f9665018700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f966009fdf0 0x7f9660139a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39678/0 (socket says 172.21.15.112:39678) 2024-09-18T11:59:59.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f9665018700 1 -- 172.21.15.112:0/2692324665 learned_addr learned my addr 172.21.15.112:0/2692324665 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:59.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.207+0000 7f9665819700 1 --1- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f96600a21c0 0x7f966014e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59494/0 (socket says 172.21.15.112:59494) 2024-09-18T11:59:59.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f9665819700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f96600a21c0 legacy=0x7f966014e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 client.?.objecter ms_handle_connect 0x7f96600a21c0 2024-09-18T11:59:59.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 client.?.objecter ms_handle_connect 0x7f966009fdf0 2024-09-18T11:59:59.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f9665018700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6789/0 conn(0x7f966009fdf0 legacy=0x7f9660139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 client.?.objecter ms_handle_connect 0x7f96600a01f0 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f9664817700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.129:6789/0 conn(0x7f96600a01f0 legacy=0x7f966014aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535914992 0 0) 0x7f966006edd0 con 0x7f966009fdf0 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): my global_id is 134214 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9644004f10 con 0x7f966009fdf0 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044240279 0 0) 0x7f966009d8d0 con 0x7f96600a01f0 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): my global_id is 144103 2024-09-18T11:59:59.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966006edd0 con 0x7f96600a01f0 2024-09-18T11:59:59.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2854631316 0 0) 0x7f9644004f10 con 0x7f966009fdf0 2024-09-18T11:59:59.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f966009d8d0 con 0x7f966009fdf0 2024-09-18T11:59:59.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9654001f00 con 0x7f966009fdf0 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): dump: 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.225 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2198851443 0 0) 0x7f966009d8d0 con 0x7f966009fdf0 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f96600a21c0 legacy=0x7f966014e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.129:6789/0 conn(0x7f96600a01f0 legacy=0x7f966014aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient: _finish_hunting 0 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 monclient: found mon.a 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660126e60 con 0x7f966009fdf0 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient: _finish_auth 0 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 5 monclient: authenticate success, global_id 134214 2024-09-18T11:59:59.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 10 monclient: _renew_subs 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9660126b80 con 0x7f966009fdf0 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 10 client.134214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 10 monclient: _renew_subs 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f966013a170 con 0x7f966009fdf0 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 1 librados: init done 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9654002390 con 0x7f966009fdf0 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f96540026a0 con 0x7f966009fdf0 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:59.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96537fe700 4 set_mon_vals no callback set 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 monclient: dump: 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f9654002930 con 0x7f966009fdf0 2024-09-18T11:59:59.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9640006590 con 0x7f9640002840 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(624..624 src has 1..624) v4 ==== 5793+0+0 (unknown 1762967044 0 0) 0x7f9654044150 con 0x7f966009fdf0 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 10 client.134214.objecter ms_dispatch 0x7f9660000fd0 osd_map(624..624 src has 1..624) v4 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 3 client.134214.objecter handle_osd_map got epochs [624,624] > 0 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 3 client.134214.objecter handle_osd_map decoding full epoch 624 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f96517fa700 20 client.134214.objecter dump_active .. 0 homeless 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.211+0000 7f9665819700 10 monclient: get_auth_request con 0x7f9644038010 auth_method 0 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.215+0000 7f96517fa700 10 client.134214.objecter ms_handle_connect 0x7f9640002840 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.215+0000 7f96517fa700 10 client.134214.objecter resend_mon_ops 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.215+0000 7f9664817700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f9640002840 legacy=0x7f9640004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.215+0000 7f96517fa700 10 client.134214.objecter ms_handle_connect 0x7f9644038010 2024-09-18T11:59:59.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748918441 0 0) 0x7f9640006590 con 0x7f9640002840 2024-09-18T11:59:59.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 10 monclient(hunting): my global_id is 104510 2024-09-18T11:59:59.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966009d8d0 con 0x7f9640002840 2024-09-18T11:59:59.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 323612031 0 0) 0x7f966009d8d0 con 0x7f9640002840 2024-09-18T11:59:59.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9640006590 con 0x7f9640002840 2024-09-18T11:59:59.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f965c002680 con 0x7f9640002840 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 10 monclient: dump: 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4190045303 0 0) 0x7f9640006590 con 0x7f9640002840 2024-09-18T11:59:59.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.411+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f964000a140 con 0x7f9640002840 2024-09-18T11:59:59.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.415+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f965c002460 con 0x7f9640002840 2024-09-18T11:59:59.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.415+0000 7f96517fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.415+0000 7f96517fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:59:59.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.415+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f9640002840 legacy=0x7f9640004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.415+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f9640002840 legacy=0x7f9640004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:59:59.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.423+0000 7f966727c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:59:59.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.423+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9640009670 con 0x7f96400090e0 2024-09-18T11:59:59.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 10 client.134214.objecter ms_handle_connect 0x7f96400090e0 2024-09-18T11:59:59.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 10 client.134214.objecter resend_mon_ops 2024-09-18T11:59:59.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f9664817700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f96400090e0 legacy=0x7f9640008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071473223 0 0) 0x7f9640009670 con 0x7f96400090e0 2024-09-18T11:59:59.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 10 monclient(hunting): my global_id is 104513 2024-09-18T11:59:59.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9640006590 con 0x7f96400090e0 2024-09-18T11:59:59.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 240361731 0 0) 0x7f9640006590 con 0x7f96400090e0 2024-09-18T11:59:59.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9640009670 con 0x7f96400090e0 2024-09-18T11:59:59.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f965c002cb0 con 0x7f96400090e0 2024-09-18T11:59:59.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:59:59.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 10 monclient: dump: 2024-09-18T11:59:59.433 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.433 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.433 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.433 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.433 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.434 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.434 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.434 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.434 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.434 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3631863798 0 0) 0x7f9640009670 con 0x7f96400090e0 2024-09-18T11:59:59.434 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.427+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9640009930 con 0x7f96400090e0 2024-09-18T11:59:59.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 2475498712) 0x7f965c0058e0 con 0x7f96400090e0 2024-09-18T11:59:59.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f96517fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:59:59.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f96517fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:59:59.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f96400090e0 legacy=0x7f9640008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f96517fa700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6790/0 conn(0x7f96400090e0 legacy=0x7f9640008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:59:59.436 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:59:59.436 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:59:59.437 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:59:59.437 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:59:59.437 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 70, 2024-09-18T11:59:59.437 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:59:59.437 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:59:59.437 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:59:59.437 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:59:59.438 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.438 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:59:59.438 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:59:59.438 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:59:59.438 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:59:59.438 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.438 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.439 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.439 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.439 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.439 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.439 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.439 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.439 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.439 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.440 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:59:59.440 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:59:59.440 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.440 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.440 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.440 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.440 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.441 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.441 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.441 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.441 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.441 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.441 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:59:59.442 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:59:59.442 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:59:59.442 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:59:59.442 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:59:59.442 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:59:59.442 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:59:59.442 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:59:59.443 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:59:59.443 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:59:59.443 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:59:59.443 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:59:59.443 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:59:59.443 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:59:59.443 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:59:59.443 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:59:59.444 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:59:59.444 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.444 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.444 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.444 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.444 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.444 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.445 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.445 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.445 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.445 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.445 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:59:59.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.445 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:59:59.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.446 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:59:59.446 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:59:59.446 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.446 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.446 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.446 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:59:59.447 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.447 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.447 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:59:59.447 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:59:59.447 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.448 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.448 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.448 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:59:59.448 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:59:59.449 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.449 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.449 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.449 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:59:59.449 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.449 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.450 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.450 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:59:59.450 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:59:59.450 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.450 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.450 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.451 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:59:59.451 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:59:59.451 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.451 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.451 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.452 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:59:59.452 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.452 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:59:59.452 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:59:59.453 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.453 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.453 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.453 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.453 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.454 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T11:59:59.455 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.456 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.456 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T11:59:59.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.456 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.456 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.456 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:59:59.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.457 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.457 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:59:59.457 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:59:59.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 10 librados: watch_flush enter 2024-09-18T11:59:59.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 10 librados: watch_flush exit 2024-09-18T11:59:59.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 20 client.134214.objecter shutdown clearing up homeless session... 2024-09-18T11:59:59.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 10 client.134214.objecter successfully canceled tick 2024-09-18T11:59:59.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.129:6812/20806 conn(0x7f9644038010 legacy=0x7f964403a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 10 monclient: shutdown 2024-09-18T11:59:59.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:59.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f9665819700 1 -- 172.21.15.112:0/2692324665 reap_dead start 2024-09-18T11:59:59.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 >> v1:172.21.15.112:6789/0 conn(0x7f966009fdf0 legacy=0x7f9660139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 shutdown_connections 2024-09-18T11:59:59.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 >> 172.21.15.112:0/2692324665 conn(0x7f96600680d0 msgr2=0x7f966009c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:59.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 shutdown_connections 2024-09-18T11:59:59.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 1 -- 172.21.15.112:0/2692324665 wait complete. 2024-09-18T11:59:59.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.431+0000 7f966727c700 1 librados: shutdown 2024-09-18T11:59:59.462 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:59:59.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T11:59:59.462 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T11:59:59.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient: get_monmap_and_config 2024-09-18T11:59:59.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient: build_initial_monmap 2024-09-18T11:59:59.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient: monmap: 2024-09-18T11:59:59.586 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:59.586 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.586 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:59:59.586216+0000 2024-09-18T11:59:59.586 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:59:59.586216+0000 2024-09-18T11:59:59.586 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:59.587 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.587 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:59.587 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:59:59.587 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:59:59.587 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 1 Processor -- start 2024-09-18T11:59:59.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 1 -- start start 2024-09-18T11:59:59.588 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient: init 2024-09-18T11:59:59.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient(hunting): picked mon.noname-b con 0x7f77880a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient(hunting): picked mon.noname-c con 0x7f7788098730 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient(hunting): picked mon.noname-a con 0x7f7788138310 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7788097af0 con 0x7f7788138310 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7788097780 con 0x7f7788098730 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778809dea0 con 0x7f77880a2040 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778759e700 10 monclient(hunting): authenticate will time out at 1771.427181s 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f77880a2040 0x7f778809d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59524/0 (socket says 172.21.15.112:59524) 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.583+0000 7f778659c700 1 -- 172.21.15.112:0/1505775722 learned_addr learned my addr 172.21.15.112:0/1505775722 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778659c700 1 -- 172.21.15.112:0/1505775722 >> v1:172.21.15.112:6790/0 conn(0x7f77880a2040 legacy=0x7f778809d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7786d9d700 1 -- 172.21.15.112:0/1505775722 >> v1:172.21.15.112:6789/0 conn(0x7f7788138310 legacy=0x7f778813a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018648996 0 0) 0x7f778809dea0 con 0x7f77880a2040 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f777dd9b700 1 -- 172.21.15.112:0/1505775722 >> v1:172.21.15.129:6789/0 conn(0x7f7788098730 legacy=0x7f778809d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient(hunting): my global_id is 104516 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774003570 con 0x7f77880a2040 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259233886 0 0) 0x7f7788097af0 con 0x7f7788138310 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient(hunting): my global_id is 134217 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778809dea0 con 0x7f7788138310 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974671489 0 0) 0x7f7774003570 con 0x7f77880a2040 2024-09-18T11:59:59.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 >> v1:172.21.15.129:6789/0 conn(0x7f7788098730 legacy=0x7f778809d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 >> v1:172.21.15.112:6789/0 conn(0x7f7788138310 legacy=0x7f778813a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: _finish_hunting 0 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 monclient: found mon.noname-b 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778809e640 con 0x7f77880a2040 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: _finish_auth 0 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7778001a00 con 0x7f77880a2040 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 5 monclient: authenticate success, global_id 104516 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T11:59:59.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: dump: 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7778001b60 con 0x7f77880a2040 2024-09-18T11:59:59.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 10 monclient: get_monmap_and_config success 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 4 set_mon_vals no callback set 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 1 -- 172.21.15.112:0/1505775722 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f77780048f0 con 0x7f77880a2040 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f7785d9b700 10 monclient: dump: 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.598 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 10 monclient: shutdown 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 1 -- 172.21.15.112:0/1505775722 >> v1:172.21.15.112:6790/0 conn(0x7f77880a2040 legacy=0x7f778809d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 1 -- 172.21.15.112:0/1505775722 shutdown_connections 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 1 -- 172.21.15.112:0/1505775722 >> 172.21.15.112:0/1505775722 conn(0x7f7788067da0 msgr2=0x7f77880a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 1 -- 172.21.15.112:0/1505775722 shutdown_connections 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 1 -- 172.21.15.112:0/1505775722 wait complete. 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 10 monclient: build_initial_monmap 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.587+0000 7f778759e700 10 monclient: monmap: 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:59:59.599 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 librados: starting msgr at 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 librados: starting objecter 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 Processor -- start 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 -- start start 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 librados: setting wanted keys 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 librados: calling monclient init 2024-09-18T11:59:59.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 10 monclient: init 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 10 monclient(hunting): picked mon.noname-b con 0x7f7788098730 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 10 monclient(hunting): picked mon.noname-a con 0x7f77880a2040 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 10 monclient(hunting): picked mon.noname-c con 0x7f7788138310 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7788096a60 con 0x7f77880a2040 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778813b140 con 0x7f7788098730 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7788135ce0 con 0x7f7788138310 2024-09-18T11:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:59.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778759e700 10 monclient(hunting): authenticate will time out at 1771.433015s 2024-09-18T11:59:59.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777dd9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f77880a2040 0x7f7788137740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39702/0 (socket says 172.21.15.112:39702) 2024-09-18T11:59:59.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777dd9b700 1 -- 172.21.15.112:0/1615765831 learned_addr learned my addr 172.21.15.112:0/1615765831 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:59.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777dd9b700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.112:6789/0 conn(0x7f77880a2040 legacy=0x7f7788137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 client.?.objecter ms_handle_connect 0x7f77880a2040 2024-09-18T11:59:59.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 client.?.objecter ms_handle_connect 0x7f7788138310 2024-09-18T11:59:59.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f7786d9d700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.112:6790/0 conn(0x7f7788138310 legacy=0x7f77881496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828478748 0 0) 0x7f7788135ce0 con 0x7f7788138310 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): my global_id is 104519 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768004f10 con 0x7f7788138310 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397898633 0 0) 0x7f7788096a60 con 0x7f77880a2040 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): my global_id is 134220 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7788135ce0 con 0x7f77880a2040 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 client.?.objecter ms_handle_connect 0x7f7788098730 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f778659c700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6789/0 conn(0x7f7788098730 legacy=0x7f7788141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739678430 0 0) 0x7f778813b140 con 0x7f7788098730 2024-09-18T11:59:59.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 10 monclient(hunting): my global_id is 144109 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.591+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7788096a60 con 0x7f7788098730 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2763352271 0 0) 0x7f7768004f10 con 0x7f7788138310 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f778813b140 con 0x7f7788138310 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7780001ef0 con 0x7f7788138310 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): dump: 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.609 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1341226092 0 0) 0x7f7788135ce0 con 0x7f77880a2040 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7768004f10 con 0x7f77880a2040 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7770001c00 con 0x7f77880a2040 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): dump: 2024-09-18T11:59:59.610 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.611 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518861088 0 0) 0x7f7788096a60 con 0x7f7788098730 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7788135ce0 con 0x7f7788098730 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7778004830 con 0x7f7788098730 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient(hunting): dump: 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2085875300 0 0) 0x7f778813b140 con 0x7f7788138310 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6789/0 conn(0x7f7788098730 legacy=0x7f7788141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.112:6789/0 conn(0x7f77880a2040 legacy=0x7f7788137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient: _finish_hunting 0 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 monclient: found mon.c 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788133e40 con 0x7f7788138310 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient: _finish_auth 0 2024-09-18T11:59:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 5 monclient: authenticate success, global_id 104519 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 10 monclient: _renew_subs 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7788135970 con 0x7f7788138310 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 10 client.104519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 10 monclient: _renew_subs 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7788141a80 con 0x7f7788138310 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 1 librados: init done 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7780002530 con 0x7f7788138310 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f77800028c0 con 0x7f7788138310 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778559a700 4 set_mon_vals no callback set 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 monclient: dump: 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f776c006590 con 0x7f776c002840 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f7780002a20 con 0x7f7788138310 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(625..625 src has 1..625) v4 ==== 5420+0+0 (unknown 1101760154 0 0) 0x7f77800440c0 con 0x7f7788138310 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 10 client.104519.objecter ms_dispatch 0x7f7788000fd0 osd_map(625..625 src has 1..625) v4 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 3 client.104519.objecter handle_osd_map got epochs [625,625] > 0 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 3 client.104519.objecter handle_osd_map decoding full epoch 625 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777effd700 20 client.104519.objecter dump_active .. 0 homeless 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.595+0000 7f777dd9b700 10 monclient: get_auth_request con 0x7f7768038530 auth_method 0 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f778659c700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6789/0 conn(0x7f776c002840 legacy=0x7f776c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 client.104519.objecter ms_handle_connect 0x7f776c002840 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 client.104519.objecter resend_mon_ops 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275612170 0 0) 0x7f776c006590 con 0x7f776c002840 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 monclient(hunting): my global_id is 144112 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778813b140 con 0x7f776c002840 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3787666088 0 0) 0x7f778813b140 con 0x7f776c002840 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f776c006590 con 0x7f776c002840 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f77780019c0 con 0x7f776c002840 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 monclient: dump: 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.617 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1268530178 0 0) 0x7f776c006590 con 0x7f776c002840 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f776c00a140 con 0x7f776c002840 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 client.104519.objecter ms_handle_connect 0x7f7768038530 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f7778005110 con 0x7f776c002840 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6789/0 conn(0x7f776c002840 legacy=0x7f776c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.599+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6789/0 conn(0x7f776c002840 legacy=0x7f776c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f778759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f776c009670 con 0x7f776c0090e0 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f778659c700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6789/0 conn(0x7f776c0090e0 legacy=0x7f776c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f777effd700 10 client.104519.objecter ms_handle_connect 0x7f776c0090e0 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f777effd700 10 client.104519.objecter resend_mon_ops 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905834172 0 0) 0x7f776c009670 con 0x7f776c0090e0 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f777effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f777effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f777effd700 10 monclient(hunting): my global_id is 144115 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.611+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f776c006590 con 0x7f776c0090e0 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1374734260 0 0) 0x7f776c006590 con 0x7f776c0090e0 2024-09-18T11:59:59.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f776c009670 con 0x7f776c0090e0 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7778005690 con 0x7f776c0090e0 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 10 monclient: dump: 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.621 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3775043845 0 0) 0x7f776c009670 con 0x7f776c0090e0 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f776c009930 con 0x7f776c0090e0 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 755370004) 0x7f7778005a20 con 0x7f776c0090e0 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6789/0 conn(0x7f776c0090e0 legacy=0x7f776c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f777effd700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6789/0 conn(0x7f776c0090e0 legacy=0x7f776c008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T11:59:59.622 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 70, 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T11:59:59.623 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.624 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T11:59:59.625 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T11:59:59.626 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T11:59:59.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.628 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T11:59:59.629 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T11:59:59.630 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T11:59:59.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T11:59:59.632 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T11:59:59.633 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T11:59:59.633 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T11:59:59.633 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T11:59:59.633 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T11:59:59.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T11:59:59.633 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T11:59:59.633 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T11:59:59.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 10 librados: watch_flush enter 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 10 librados: watch_flush exit 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 20 client.104519.objecter shutdown clearing up homeless session... 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 10 client.104519.objecter successfully canceled tick 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.129:6812/20806 conn(0x7f7768038530 legacy=0x7f776803a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 10 monclient: shutdown 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 >> v1:172.21.15.112:6790/0 conn(0x7f7788138310 legacy=0x7f77881496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f7786d9d700 1 -- 172.21.15.112:0/1615765831 reap_dead start 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 shutdown_connections 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 >> 172.21.15.112:0/1615765831 conn(0x7f7788067da0 msgr2=0x7f778813b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 shutdown_connections 2024-09-18T11:59:59.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 1 -- 172.21.15.112:0/1615765831 wait complete. 2024-09-18T11:59:59.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.615+0000 7f778759e700 1 librados: shutdown 2024-09-18T11:59:59.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T11:59:59.636 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T11:59:59.636 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T11:59:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient: get_monmap_and_config 2024-09-18T11:59:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient: build_initial_monmap 2024-09-18T11:59:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient: monmap: 2024-09-18T11:59:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:59:59.745953+0000 2024-09-18T11:59:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:59:59.745953+0000 2024-09-18T11:59:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 1 Processor -- start 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 1 -- start start 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient: init 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient(hunting): picked mon.noname-b con 0x7f1bcc0a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc12edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc12f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc0981a0 con 0x7f1bcc12edc0 2024-09-18T11:59:59.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc06c120 con 0x7f1bcc12f8a0 2024-09-18T11:59:59.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc06eef0 con 0x7f1bcc0a2040 2024-09-18T11:59:59.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:59.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd2b93700 10 monclient(hunting): authenticate will time out at 1771.586784s 2024-09-18T11:59:59.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd1390700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1bcc12edc0 0x7f1bcc12f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39710/0 (socket says 172.21.15.112:39710) 2024-09-18T11:59:59.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.743+0000 7f1bd1390700 1 -- 172.21.15.112:0/2835609446 learned_addr learned my addr 172.21.15.112:0/2835609446 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd1390700 1 -- 172.21.15.112:0/2835609446 >> v1:172.21.15.112:6789/0 conn(0x7f1bcc12edc0 legacy=0x7f1bcc12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd1b91700 1 -- 172.21.15.112:0/2835609446 >> v1:172.21.15.112:6790/0 conn(0x7f1bcc0a2040 legacy=0x7f1bcc097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2392700 1 -- 172.21.15.112:0/2835609446 >> v1:172.21.15.129:6789/0 conn(0x7f1bcc12f8a0 legacy=0x7f1bcc137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557704537 0 0) 0x7f1bcc0981a0 con 0x7f1bcc12edc0 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2392700 1 -- 172.21.15.112:0/2835609446 >> v1:172.21.15.129:6789/0 conn(0x7f1bcc12f8a0 legacy=0x7f1bcc137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2392700 1 -- 172.21.15.112:0/2835609446 >> v1:172.21.15.129:6789/0 conn(0x7f1bcc12f8a0 legacy=0x7f1bcc137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2392700 1 --1- 172.21.15.112:0/2835609446 >> v1:172.21.15.129:6789/0 conn(0x7f1bcc12f8a0 0x7f1bcc137230 :-1 s=OPENED pgs=9180 cs=1 l=1).handle_message read tag failed 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2392700 1 --1- 172.21.15.112:0/2835609446 >> v1:172.21.15.129:6789/0 conn(0x7f1bcc12f8a0 0x7f1bcc137230 :-1 s=OPENED pgs=9180 cs=1 l=1).fault on lossy channel, failing 2024-09-18T11:59:59.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient(hunting): my global_id is 134223 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb4003570 con 0x7f1bcc12edc0 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.129:6789/0 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231560418 0 0) 0x7f1bcc06eef0 con 0x7f1bcc0a2040 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient(hunting): my global_id is 104522 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bcc0981a0 con 0x7f1bcc0a2040 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3646279922 0 0) 0x7f1bb4003570 con 0x7f1bcc12edc0 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 >> v1:172.21.15.112:6790/0 conn(0x7f1bcc0a2040 legacy=0x7f1bcc097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 >> v1:172.21.15.129:6789/0 conn(0x7f1bcc12f8a0 legacy=0x7f1bcc137230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: _finish_hunting 0 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 monclient: found mon.noname-a 2024-09-18T11:59:59.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc096dd0 con 0x7f1bcc12edc0 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: _finish_auth 0 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 5 monclient: authenticate success, global_id 134223 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1bc4001940 con 0x7f1bcc12edc0 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: dump: 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.752 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc4001bd0 con 0x7f1bcc12edc0 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 10 monclient: get_monmap_and_config success 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 4 set_mon_vals no callback set 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 1 -- 172.21.15.112:0/2835609446 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1bc4004a90 con 0x7f1bcc12edc0 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd0b8f700 10 monclient: dump: 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.754 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 10 monclient: shutdown 2024-09-18T11:59:59.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T11:59:59.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 1 -- 172.21.15.112:0/2835609446 >> v1:172.21.15.112:6789/0 conn(0x7f1bcc12edc0 legacy=0x7f1bcc12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 1 -- 172.21.15.112:0/2835609446 shutdown_connections 2024-09-18T11:59:59.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 1 -- 172.21.15.112:0/2835609446 >> 172.21.15.112:0/2835609446 conn(0x7f1bcc067da0 msgr2=0x7f1bcc0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 1 -- 172.21.15.112:0/2835609446 shutdown_connections 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.747+0000 7f1bd2b93700 1 -- 172.21.15.112:0/2835609446 wait complete. 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient: build_initial_monmap 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient: monmap: 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T11:59:59.758 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 librados: starting msgr at 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 librados: starting objecter 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 Processor -- start 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 -- start start 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 librados: setting wanted keys 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 librados: calling monclient init 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient: init 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient: _reopen_session rank -1 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc0a2040 addr v1:172.21.15.112:6790/0 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc12edc0 addr v1:172.21.15.112:6789/0 2024-09-18T11:59:59.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient(hunting): picked mon.noname-b con 0x7f1bcc12f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc06f110 con 0x7f1bcc12edc0 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc0a0170 con 0x7f1bcc12f8a0 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc140030 con 0x7f1bcc0a2040 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient(hunting): _renew_subs 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd2b93700 10 monclient(hunting): authenticate will time out at 1771.594913s 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd1b91700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1bcc0a2040 0x7f1bcc13f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59550/0 (socket says 172.21.15.112:59550) 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.751+0000 7f1bd1b91700 1 -- 172.21.15.112:0/1567727147 learned_addr learned my addr 172.21.15.112:0/1567727147 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bd1b91700 1 -- 172.21.15.112:0/1567727147 >> v1:172.21.15.112:6790/0 conn(0x7f1bcc0a2040 legacy=0x7f1bcc13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 client.?.objecter ms_handle_connect 0x7f1bcc0a2040 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 client.?.objecter ms_handle_connect 0x7f1bcc12edc0 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bd1390700 1 -- 172.21.15.112:0/1567727147 >> v1:172.21.15.112:6789/0 conn(0x7f1bcc12edc0 legacy=0x7f1bcc13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44923256 0 0) 0x7f1bcc140030 con 0x7f1bcc0a2040 2024-09-18T11:59:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): my global_id is 104525 2024-09-18T11:59:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb8004f10 con 0x7f1bcc0a2040 2024-09-18T11:59:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bd2392700 1 -- 172.21.15.112:0/1567727147 >> v1:172.21.15.129:6789/0 conn(0x7f1bcc12f8a0 legacy=0x7f1bcc1477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T11:59:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 DispatchQueue::entry inject delay of 0.002000 2024-09-18T11:59:59.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 client.?.objecter ms_handle_connect 0x7f1bcc12f8a0 2024-09-18T11:59:59.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T11:59:59.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818630078 0 0) 0x7f1bcc06f110 con 0x7f1bcc12edc0 2024-09-18T11:59:59.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): my global_id is 134226 2024-09-18T11:59:59.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bcc140030 con 0x7f1bcc12edc0 2024-09-18T11:59:59.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557718629 0 0) 0x7f1bcc0a0170 con 0x7f1bcc12f8a0 2024-09-18T11:59:59.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T11:59:59.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T11:59:59.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): my global_id is 144118 2024-09-18T11:59:59.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bcc06f110 con 0x7f1bcc12f8a0 2024-09-18T11:59:59.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2357337363 0 0) 0x7f1bb8004f10 con 0x7f1bcc0a2040 2024-09-18T11:59:59.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bcc0a0170 con 0x7f1bcc0a2040 2024-09-18T11:59:59.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1bc0001f00 con 0x7f1bcc0a2040 2024-09-18T11:59:59.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T11:59:59.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): dump: 2024-09-18T11:59:59.767 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.768 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.768 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.769 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.755+0000 7f1bc9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2537393726 0 0) 0x7f1bcc0a0170 con 0x7f1bcc0a2040 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 >> v1:172.21.15.129:6789/0 conn(0x7f1bcc12f8a0 legacy=0x7f1bcc1477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 >> v1:172.21.15.112:6789/0 conn(0x7f1bcc12edc0 legacy=0x7f1bcc13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 10 monclient: _finish_hunting 0 2024-09-18T11:59:59.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 monclient: found mon.c 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc136250 con 0x7f1bcc0a2040 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 10 monclient: _finish_auth 0 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 5 monclient: authenticate success, global_id 104525 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 10 monclient: _renew_subs 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc137140 con 0x7f1bcc0a2040 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 10 client.104525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 10 monclient: _renew_subs 2024-09-18T11:59:59.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc06c7f0 con 0x7f1bcc0a2040 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 1 librados: init done 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc0001940 con 0x7f1bcc0a2040 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bcbfff700 4 set_mon_vals no callback set 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1bc0002560 con 0x7f1bcc0a2040 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 10 monclient: dump: 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.772 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f1bc0002740 con 0x7f1bcc0a2040 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac0050b0 con 0x7f1bcc0a2040 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(625..625 src has 1..625) v4 ==== 5420+0+0 (unknown 1101760154 0 0) 0x7f1bc0043f50 con 0x7f1bcc0a2040 2024-09-18T11:59:59.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 10 client.104525.objecter ms_dispatch 0x7f1bcc000fd0 osd_map(625..625 src has 1..625) v4 2024-09-18T11:59:59.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 3 client.104525.objecter handle_osd_map got epochs [625,625] > 0 2024-09-18T11:59:59.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 3 client.104525.objecter handle_osd_map decoding full epoch 625 2024-09-18T11:59:59.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bc9ffb700 20 client.104525.objecter dump_active .. 0 homeless 2024-09-18T11:59:59.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.759+0000 7f1bd1390700 10 monclient: get_auth_request con 0x7f1bb8038400 auth_method 0 2024-09-18T11:59:59.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.763+0000 7f1bc9ffb700 10 client.104525.objecter ms_handle_connect 0x7f1bb8038400 2024-09-18T11:59:59.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.767+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f1bc0014b00 con 0x7f1bcc0a2040 2024-09-18T11:59:59.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.767+0000 7f1bc9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T11:59:59.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.767+0000 7f1bc9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:00.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.999+0000 7f1bd2b93700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:00:00.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.999+0000 7f1bd2b93700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:00:00.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.999+0000 7f1bd2b93700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:00:00.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T11:59:59.999+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1bac005c30 con 0x7f1bcc0a2040 2024-09-18T12:00:00.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.003+0000 7f1bc9ffb700 1 -- 172.21.15.112:0/1567727147 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f1bc0014500 con 0x7f1bcc0a2040 2024-09-18T12:00:00.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.003+0000 7f1bc9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:00:00.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.003+0000 7f1bc9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:00.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 10 librados: watch_flush enter 2024-09-18T12:00:00.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 10 librados: watch_flush exit 2024-09-18T12:00:00.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 20 client.104525.objecter shutdown clearing up homeless session... 2024-09-18T12:00:00.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 10 client.104525.objecter successfully canceled tick 2024-09-18T12:00:00.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 >> v1:172.21.15.129:6812/20806 conn(0x7f1bb8038400 legacy=0x7f1bb803a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:00.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 10 monclient: shutdown 2024-09-18T12:00:00.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:00.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 >> v1:172.21.15.112:6790/0 conn(0x7f1bcc0a2040 legacy=0x7f1bcc13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:00.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 shutdown_connections 2024-09-18T12:00:00.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 >> 172.21.15.112:0/1567727147 conn(0x7f1bcc067da0 msgr2=0x7f1bcc06ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:00.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 shutdown_connections 2024-09-18T12:00:00.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 1 -- 172.21.15.112:0/1567727147 wait complete. 2024-09-18T12:00:00.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:00.007+0000 7f1bd2b93700 1 librados: shutdown 2024-09-18T12:00:00.015 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.011+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.015 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.011+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=1,health=10,kv=8,logm=59} crc {auth=1124120954,config=1228700967,health=2307001552,kv=1310296871,logm=204106762}) 2024-09-18T12:00:00.015 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.011+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=11,kv=7,logm=59} crc {auth=3404792002,config=3140514009,health=1938870710,kv=2459836832,logm=1927326934}) 2024-09-18T12:00:00.015 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.011+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.015 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.011+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=1,health=10,kv=8,logm=59} crc {auth=1124120954,config=1228700967,health=2307001552,kv=1310296871,logm=204106762}) 2024-09-18T12:00:00.015 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.011+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=7,logm=59} crc {auth=1811005891,config=3140514009,health=1337383348,kv=2459836832,logm=1276225894}) 2024-09-18T12:00:00.017 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.015+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.017 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.015+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257998290}) 2024-09-18T12:00:00.017 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.015+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1887500885}) 2024-09-18T12:00:00.017 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.015+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.017 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.015+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257998290}) 2024-09-18T12:00:00.017 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.015+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3476440207}) 2024-09-18T12:00:00.020 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.020 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665748601}) 2024-09-18T12:00:00.020 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1321390956}) 2024-09-18T12:00:00.020 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.020 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665748601}) 2024-09-18T12:00:00.020 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2103262768}) 2024-09-18T12:00:00.022 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.022 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213556208}) 2024-09-18T12:00:00.022 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057918416}) 2024-09-18T12:00:00.022 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.022 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213556208}) 2024-09-18T12:00:00.022 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.019+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926801351}) 2024-09-18T12:00:00.024 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.024 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628683719}) 2024-09-18T12:00:00.024 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1473215160}) 2024-09-18T12:00:00.024 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.024 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628683719}) 2024-09-18T12:00:00.024 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=260930678}) 2024-09-18T12:00:00.025 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:00:00.026 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.026 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940590937}) 2024-09-18T12:00:00.026 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909452538}) 2024-09-18T12:00:00.026 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.026 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940590937}) 2024-09-18T12:00:00.027 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.023+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398787319}) 2024-09-18T12:00:00.027 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.027 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=335495754}) 2024-09-18T12:00:00.028 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=969114973}) 2024-09-18T12:00:00.028 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.028 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=335495754}) 2024-09-18T12:00:00.028 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398638619}) 2024-09-18T12:00:00.028 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.028 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645458705}) 2024-09-18T12:00:00.029 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3252107688}) 2024-09-18T12:00:00.029 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.029 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645458705}) 2024-09-18T12:00:00.029 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2462782474}) 2024-09-18T12:00:00.030 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.030 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345333921}) 2024-09-18T12:00:00.030 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3350472933}) 2024-09-18T12:00:00.030 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.030 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345333921}) 2024-09-18T12:00:00.030 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.027+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2102532271}) 2024-09-18T12:00:00.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981712043}) 2024-09-18T12:00:00.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1389831637}) 2024-09-18T12:00:00.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981712043}) 2024-09-18T12:00:00.031 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2407772869}) 2024-09-18T12:00:00.032 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.032 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462658513}) 2024-09-18T12:00:00.032 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895861089}) 2024-09-18T12:00:00.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462658513}) 2024-09-18T12:00:00.033 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=861630372}) 2024-09-18T12:00:00.034 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.034 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801858788}) 2024-09-18T12:00:00.034 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=63323160}) 2024-09-18T12:00:00.034 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.034 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801858788}) 2024-09-18T12:00:00.034 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.031+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172728215}) 2024-09-18T12:00:00.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592983515}) 2024-09-18T12:00:00.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821687170}) 2024-09-18T12:00:00.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592983515}) 2024-09-18T12:00:00.035 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2724744116}) 2024-09-18T12:00:00.036 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.036 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984798353}) 2024-09-18T12:00:00.036 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1292066042}) 2024-09-18T12:00:00.036 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.036 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984798353}) 2024-09-18T12:00:00.036 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1561055729}) 2024-09-18T12:00:00.037 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.037 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=681166583}) 2024-09-18T12:00:00.038 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2888822226}) 2024-09-18T12:00:00.038 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.038 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=681166583}) 2024-09-18T12:00:00.038 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3834221176}) 2024-09-18T12:00:00.039 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.035+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.039 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014308690}) 2024-09-18T12:00:00.039 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639483903}) 2024-09-18T12:00:00.039 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.039 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014308690}) 2024-09-18T12:00:00.039 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=812298570}) 2024-09-18T12:00:00.040 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.040 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050615615}) 2024-09-18T12:00:00.040 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018986968}) 2024-09-18T12:00:00.040 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.040 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050615615}) 2024-09-18T12:00:00.040 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2142282016}) 2024-09-18T12:00:00.041 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.041 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193578897}) 2024-09-18T12:00:00.041 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1521654000}) 2024-09-18T12:00:00.041 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.041 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193578897}) 2024-09-18T12:00:00.042 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2513095350}) 2024-09-18T12:00:00.043 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.043 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688416613}) 2024-09-18T12:00:00.043 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2305658918}) 2024-09-18T12:00:00.043 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.043 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688416613}) 2024-09-18T12:00:00.043 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.039+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185276510}) 2024-09-18T12:00:00.044 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.044 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165960571}) 2024-09-18T12:00:00.044 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3039785487}) 2024-09-18T12:00:00.044 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.045 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165960571}) 2024-09-18T12:00:00.045 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3069326873}) 2024-09-18T12:00:00.045 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.045 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298168644}) 2024-09-18T12:00:00.045 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2132635970}) 2024-09-18T12:00:00.045 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.045 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298168644}) 2024-09-18T12:00:00.045 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3218491523}) 2024-09-18T12:00:00.046 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.046 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1348659381}) 2024-09-18T12:00:00.046 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=966968474}) 2024-09-18T12:00:00.046 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.046 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1348659381}) 2024-09-18T12:00:00.046 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3297314531}) 2024-09-18T12:00:00.047 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.047 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269241346}) 2024-09-18T12:00:00.047 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984065620}) 2024-09-18T12:00:00.047 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.047 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269241346}) 2024-09-18T12:00:00.047 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.043+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1995781572}) 2024-09-18T12:00:00.048 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.048 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=927485079}) 2024-09-18T12:00:00.048 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2248921375}) 2024-09-18T12:00:00.048 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.048 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=927485079}) 2024-09-18T12:00:00.048 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=315645652}) 2024-09-18T12:00:00.049 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.049 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047663897}) 2024-09-18T12:00:00.049 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3825588415}) 2024-09-18T12:00:00.049 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.049 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047663897}) 2024-09-18T12:00:00.049 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.047+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783617347}) 2024-09-18T12:00:00.188 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.188 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4003585509}) 2024-09-18T12:00:00.188 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266188764}) 2024-09-18T12:00:00.188 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.188 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4003585509}) 2024-09-18T12:00:00.188 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581420986}) 2024-09-18T12:00:00.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452587075}) 2024-09-18T12:00:00.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2938099425}) 2024-09-18T12:00:00.190 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.191 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452587075}) 2024-09-18T12:00:00.191 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.187+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2889383086}) 2024-09-18T12:00:00.193 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.193 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118617742}) 2024-09-18T12:00:00.193 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3714708779}) 2024-09-18T12:00:00.193 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.193 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118617742}) 2024-09-18T12:00:00.193 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965165612}) 2024-09-18T12:00:00.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577797962}) 2024-09-18T12:00:00.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3450162459}) 2024-09-18T12:00:00.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577797962}) 2024-09-18T12:00:00.195 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.191+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973163985}) 2024-09-18T12:00:00.197 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.195+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.197 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.195+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008832438}) 2024-09-18T12:00:00.197 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.195+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=158033695}) 2024-09-18T12:00:00.197 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.195+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.197 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.195+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008832438}) 2024-09-18T12:00:00.197 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.195+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413135107}) 2024-09-18T12:00:00.199 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.199 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840377749}) 2024-09-18T12:00:00.199 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3306898430}) 2024-09-18T12:00:00.200 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.200 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840377749}) 2024-09-18T12:00:00.200 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3414125591}) 2024-09-18T12:00:00.201 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.201 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1822733795}) 2024-09-18T12:00:00.201 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939784143}) 2024-09-18T12:00:00.201 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.202 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1822733795}) 2024-09-18T12:00:00.202 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596720174}) 2024-09-18T12:00:00.203 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.203 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232727037}) 2024-09-18T12:00:00.203 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837347710}) 2024-09-18T12:00:00.203 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.203 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232727037}) 2024-09-18T12:00:00.203 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.199+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644821219}) 2024-09-18T12:00:00.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074136988}) 2024-09-18T12:00:00.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135304090}) 2024-09-18T12:00:00.204 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.205 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074136988}) 2024-09-18T12:00:00.205 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817487133}) 2024-09-18T12:00:00.206 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.206 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806310933}) 2024-09-18T12:00:00.206 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3873998452}) 2024-09-18T12:00:00.206 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.206 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806310933}) 2024-09-18T12:00:00.206 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.203+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3296757573}) 2024-09-18T12:00:00.207 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.207 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737437179}) 2024-09-18T12:00:00.207 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884255397}) 2024-09-18T12:00:00.208 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.208 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737437179}) 2024-09-18T12:00:00.208 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3328955085}) 2024-09-18T12:00:00.209 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.209 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1935552950}) 2024-09-18T12:00:00.209 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4148572197}) 2024-09-18T12:00:00.209 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.210 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1935552950}) 2024-09-18T12:00:00.210 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268966293}) 2024-09-18T12:00:00.211 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.211 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3656706266}) 2024-09-18T12:00:00.211 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4070231961}) 2024-09-18T12:00:00.211 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.211 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3656706266}) 2024-09-18T12:00:00.211 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.207+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3634272645}) 2024-09-18T12:00:00.212 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.213 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818826652}) 2024-09-18T12:00:00.213 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3034911235}) 2024-09-18T12:00:00.213 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.213 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818826652}) 2024-09-18T12:00:00.213 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=632747391}) 2024-09-18T12:00:00.215 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.215 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825934905}) 2024-09-18T12:00:00.216 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3604451835}) 2024-09-18T12:00:00.216 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.216 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825934905}) 2024-09-18T12:00:00.216 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.211+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=16373943}) 2024-09-18T12:00:00.216 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.216 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535434826}) 2024-09-18T12:00:00.217 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=121903186}) 2024-09-18T12:00:00.217 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.217 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535434826}) 2024-09-18T12:00:00.217 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990505242}) 2024-09-18T12:00:00.219 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.219 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015538998}) 2024-09-18T12:00:00.219 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552439905}) 2024-09-18T12:00:00.219 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.219 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015538998}) 2024-09-18T12:00:00.220 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620265588}) 2024-09-18T12:00:00.220 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.220 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337983611}) 2024-09-18T12:00:00.220 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2337766715}) 2024-09-18T12:00:00.220 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.220 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337983611}) 2024-09-18T12:00:00.221 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.215+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1124294000}) 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148191728}) 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1335264974}) 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148191728}) 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3711403577}) 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1619730842}) 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043676465}) 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.224 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1619730842}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1020285541}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718181126}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2418766106}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718181126}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.219+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1281943573}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70904298}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495846132}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70904298}) 2024-09-18T12:00:00.225 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=781180164}) 2024-09-18T12:00:00.227 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.227 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107187888}) 2024-09-18T12:00:00.227 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2016633310}) 2024-09-18T12:00:00.227 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.227 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107187888}) 2024-09-18T12:00:00.227 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322389918}) 2024-09-18T12:00:00.228 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.228 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663648956}) 2024-09-18T12:00:00.228 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005050632}) 2024-09-18T12:00:00.228 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.228 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663648956}) 2024-09-18T12:00:00.228 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=654404717}) 2024-09-18T12:00:00.229 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.229 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799840219}) 2024-09-18T12:00:00.229 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=862670625}) 2024-09-18T12:00:00.229 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.229 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799840219}) 2024-09-18T12:00:00.229 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.223+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887744777}) 2024-09-18T12:00:00.233 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.234 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293136852}) 2024-09-18T12:00:00.234 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2903118622}) 2024-09-18T12:00:00.234 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.234 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293136852}) 2024-09-18T12:00:00.234 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3959958935}) 2024-09-18T12:00:00.234 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.234 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244727783}) 2024-09-18T12:00:00.235 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1745602726}) 2024-09-18T12:00:00.235 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.235 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244727783}) 2024-09-18T12:00:00.235 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1499441376}) 2024-09-18T12:00:00.235 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.235 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462638152}) 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4187335049}) 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462638152}) 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.227+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1722581118}) 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649525014}) 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3078708018}) 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649525014}) 2024-09-18T12:00:00.236 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94108900}) 2024-09-18T12:00:00.237 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.237 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462904699}) 2024-09-18T12:00:00.237 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543453857}) 2024-09-18T12:00:00.237 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.237 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462904699}) 2024-09-18T12:00:00.237 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168943158}) 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264540408}) 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639654587}) 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264540408}) 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.231+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256306286}) 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.235+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.235+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256291841}) 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.235+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2083388573}) 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.235+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.235+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256291841}) 2024-09-18T12:00:00.239 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.235+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=115420687}) 2024-09-18T12:00:00.250 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.247+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.250 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.247+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472058407}) 2024-09-18T12:00:00.250 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.247+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755230751}) 2024-09-18T12:00:00.250 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.247+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.250 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.247+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472058407}) 2024-09-18T12:00:00.251 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.247+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815349704}) 2024-09-18T12:00:00.252 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.252 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209497516}) 2024-09-18T12:00:00.252 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1562396226}) 2024-09-18T12:00:00.252 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.252 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209497516}) 2024-09-18T12:00:00.252 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3120803098}) 2024-09-18T12:00:00.254 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.254 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3781722048,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=674701895}) 2024-09-18T12:00:00.254 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=2108073348,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=3045963548}) 2024-09-18T12:00:00.254 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.254 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3781722048,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=674701895}) 2024-09-18T12:00:00.255 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.251+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2876953531,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=2313653941}) 2024-09-18T12:00:00.256 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.256 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=37} crc {mgrstat=1845491019,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=104099530,osd_snap=1796356724,osdmap=3288120013}) 2024-09-18T12:00:00.256 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=44,osdmap=37} crc {mgrstat=1845491019,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_snap=2750003713,osdmap=794807134}) 2024-09-18T12:00:00.256 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.257 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=37} crc {mgrstat=1845491019,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=104099530,osd_snap=1796356724,osdmap=3288120013}) 2024-09-18T12:00:00.257 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=5,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=37} crc {mgrstat=1845491019,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=104099530,osd_snap=1655813147,osdmap=2402257638}) 2024-09-18T12:00:00.258 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.258 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3602754403}) 2024-09-18T12:00:00.259 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3728272313}) 2024-09-18T12:00:00.259 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.259 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3602754403}) 2024-09-18T12:00:00.259 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.255+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=336359284}) 2024-09-18T12:00:00.261 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.261 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=251834082}) 2024-09-18T12:00:00.261 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2797260075}) 2024-09-18T12:00:00.261 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.261 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=251834082}) 2024-09-18T12:00:00.261 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1460768855}) 2024-09-18T12:00:00.262 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.262 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=79058587}) 2024-09-18T12:00:00.263 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3770819944}) 2024-09-18T12:00:00.263 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.263 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=79058587}) 2024-09-18T12:00:00.263 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.259+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3763767153}) 2024-09-18T12:00:00.264 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.264 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1927449327}) 2024-09-18T12:00:00.264 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=314307427}) 2024-09-18T12:00:00.264 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.264 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1927449327}) 2024-09-18T12:00:00.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3889271667}) 2024-09-18T12:00:00.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2824791088}) 2024-09-18T12:00:00.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=680067624}) 2024-09-18T12:00:00.266 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.266 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2824791088}) 2024-09-18T12:00:00.266 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.263+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2519008766}) 2024-09-18T12:00:00.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3066057108}) 2024-09-18T12:00:00.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3750239118}) 2024-09-18T12:00:00.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3066057108}) 2024-09-18T12:00:00.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2298102390}) 2024-09-18T12:00:00.270 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.270 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1153699911}) 2024-09-18T12:00:00.270 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=616109766}) 2024-09-18T12:00:00.270 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.271 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1153699911}) 2024-09-18T12:00:00.271 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.267+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3672395185}) 2024-09-18T12:00:00.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.271+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.271+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2418794725}) 2024-09-18T12:00:00.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.271+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=922119382}) 2024-09-18T12:00:00.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.271+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.273 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.271+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2418794725}) 2024-09-18T12:00:00.273 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.271+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2103526016}) 2024-09-18T12:00:00.279 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.275+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.279 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.275+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=75} crc {osdmap=3443631652}) 2024-09-18T12:00:00.279 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.275+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=71} crc {osdmap=1890838074}) 2024-09-18T12:00:00.280 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.275+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:00:00.280 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.275+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=75} crc {osdmap=3443631652}) 2024-09-18T12:00:00.280 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:00:00.275+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=73} crc {osdmap=3672555041}) 2024-09-18T12:00:01.026 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:00:01.026 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:00:01.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.135+0000 7f4a1f59e700 10 monclient: get_monmap_and_config 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.135+0000 7f4a1f59e700 10 monclient: build_initial_monmap 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.135+0000 7f4a1f59e700 10 monclient: monmap: 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:01.138986+0000 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:01.138986+0000 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:01.138 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.135+0000 7f4a1f59e700 1 Processor -- start 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.135+0000 7f4a1f59e700 1 -- start start 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.135+0000 7f4a1f59e700 10 monclient: init 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f4a200a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f4a200a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f4a20097970 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a20097550 con 0x7f4a20097970 2024-09-18T12:00:01.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a20096cb0 con 0x7f4a200a0e60 2024-09-18T12:00:01.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a200a1440 con 0x7f4a200a0380 2024-09-18T12:00:01.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:01.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 10 monclient(hunting): authenticate will time out at 1772.980505s 2024-09-18T12:00:01.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1e59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f4a200a0380 0x7f4a200a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59554/0 (socket says 172.21.15.112:59554) 2024-09-18T12:00:01.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1e59c700 1 -- 172.21.15.112:0/452088730 learned_addr learned my addr 172.21.15.112:0/452088730 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:01.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1e59c700 1 -- 172.21.15.112:0/452088730 >> v1:172.21.15.112:6790/0 conn(0x7f4a200a0380 legacy=0x7f4a200a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1ed9d700 1 -- 172.21.15.112:0/452088730 >> v1:172.21.15.112:6789/0 conn(0x7f4a20097970 legacy=0x7f4a2013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548723284 0 0) 0x7f4a200a1440 con 0x7f4a200a0380 2024-09-18T12:00:01.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): my global_id is 104528 2024-09-18T12:00:01.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a15d9b700 1 -- 172.21.15.112:0/452088730 >> v1:172.21.15.129:6789/0 conn(0x7f4a200a0e60 legacy=0x7f4a20137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a0c003570 con 0x7f4a200a0380 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333662861 0 0) 0x7f4a20097550 con 0x7f4a20097970 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): my global_id is 134229 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a200a1440 con 0x7f4a20097970 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148402330 0 0) 0x7f4a20096cb0 con 0x7f4a200a0e60 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient(hunting): my global_id is 144121 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a20097550 con 0x7f4a200a0e60 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3157222066 0 0) 0x7f4a0c003570 con 0x7f4a200a0380 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 >> v1:172.21.15.129:6789/0 conn(0x7f4a200a0e60 legacy=0x7f4a20137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 >> v1:172.21.15.112:6789/0 conn(0x7f4a20097970 legacy=0x7f4a2013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient: _finish_hunting 0 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 monclient: found mon.noname-b 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a200a1de0 con 0x7f4a200a0380 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient: _finish_auth 0 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 5 monclient: authenticate success, global_id 104528 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4a10001940 con 0x7f4a200a0380 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.139+0000 7f4a1dd9b700 10 monclient: dump: 2024-09-18T12:00:01.146 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a10001bd0 con 0x7f4a200a0380 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1dd9b700 1 -- 172.21.15.112:0/452088730 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4a10004a90 con 0x7f4a200a0380 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 4 set_mon_vals no callback set 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1dd9b700 10 monclient: dump: 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.148 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 10 monclient: shutdown 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 -- 172.21.15.112:0/452088730 >> v1:172.21.15.112:6790/0 conn(0x7f4a200a0380 legacy=0x7f4a200a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 -- 172.21.15.112:0/452088730 shutdown_connections 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 -- 172.21.15.112:0/452088730 >> 172.21.15.112:0/452088730 conn(0x7f4a20068100 msgr2=0x7f4a200a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 -- 172.21.15.112:0/452088730 shutdown_connections 2024-09-18T12:00:01.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 -- 172.21.15.112:0/452088730 wait complete. 2024-09-18T12:00:01.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 10 monclient: build_initial_monmap 2024-09-18T12:00:01.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 10 monclient: monmap: 2024-09-18T12:00:01.153 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:01.153 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:01.153 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 librados: starting msgr at 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 librados: starting objecter 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 Processor -- start 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 -- start start 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 librados: setting wanted keys 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 1 librados: calling monclient init 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.143+0000 7f4a1f59e700 10 monclient: init 2024-09-18T12:00:01.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f4a20097970 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f4a200a0380 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f4a200a0e60 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a201219e0 con 0x7f4a200a0380 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a20121670 con 0x7f4a20097970 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a20142480 con 0x7f4a200a0e60 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1f59e700 10 monclient(hunting): authenticate will time out at 1772.988934s 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a15d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f4a200a0380 0x7f4a20146920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39732/0 (socket says 172.21.15.112:39732) 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a15d9b700 1 -- 172.21.15.112:0/3729723972 learned_addr learned my addr 172.21.15.112:0/3729723972 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a15d9b700 1 -- 172.21.15.112:0/3729723972 >> v1:172.21.15.112:6789/0 conn(0x7f4a200a0380 legacy=0x7f4a20146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 client.?.objecter ms_handle_connect 0x7f4a200a0380 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 client.?.objecter ms_handle_connect 0x7f4a200a0e60 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1ed9d700 1 -- 172.21.15.112:0/3729723972 >> v1:172.21.15.112:6790/0 conn(0x7f4a200a0e60 legacy=0x7f4a2014a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 client.?.objecter ms_handle_connect 0x7f4a20097970 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a1e59c700 1 -- 172.21.15.112:0/3729723972 >> v1:172.21.15.129:6789/0 conn(0x7f4a20097970 legacy=0x7f4a20134270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877224128 0 0) 0x7f4a201219e0 con 0x7f4a200a0380 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): my global_id is 134232 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a00004f10 con 0x7f4a200a0380 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994606640 0 0) 0x7f4a20142480 con 0x7f4a200a0e60 2024-09-18T12:00:01.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): my global_id is 104531 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a201219e0 con 0x7f4a200a0e60 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616001058 0 0) 0x7f4a20121670 con 0x7f4a20097970 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 10 monclient(hunting): my global_id is 144124 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.147+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a20142480 con 0x7f4a20097970 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3137404552 0 0) 0x7f4a00004f10 con 0x7f4a200a0380 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a20121670 con 0x7f4a200a0380 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4a08001d10 con 0x7f4a200a0380 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): dump: 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2375847071 0 0) 0x7f4a201219e0 con 0x7f4a200a0e60 2024-09-18T12:00:01.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a00004f10 con 0x7f4a200a0e60 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 879635096 0 0) 0x7f4a20142480 con 0x7f4a20097970 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a201219e0 con 0x7f4a20097970 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4a18001f40 con 0x7f4a200a0e60 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): dump: 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.159 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.160 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.160 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4a10001360 con 0x7f4a20097970 2024-09-18T12:00:01.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:00:01.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient(hunting): dump: 2024-09-18T12:00:01.160 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.160 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2701511791 0 0) 0x7f4a20121670 con 0x7f4a200a0380 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 >> v1:172.21.15.112:6790/0 conn(0x7f4a200a0e60 legacy=0x7f4a2014a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 >> v1:172.21.15.129:6789/0 conn(0x7f4a20097970 legacy=0x7f4a20134270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 monclient: found mon.a 2024-09-18T12:00:01.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a20121390 con 0x7f4a200a0380 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient: _finish_auth 0 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 5 monclient: authenticate success, global_id 134232 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 10 monclient: _renew_subs 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2011f860 con 0x7f4a200a0380 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a080022b0 con 0x7f4a200a0380 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 10 client.134232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4a08002600 con 0x7f4a200a0380 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:01.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 10 monclient: dump: 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1cd99700 4 set_mon_vals no callback set 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 10 monclient: _renew_subs 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a20136c60 con 0x7f4a200a0380 2024-09-18T12:00:01.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 1 librados: init done 2024-09-18T12:00:01.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a201205a0 con 0x7f4a200a0380 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.151+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f4a080027e0 con 0x7f4a200a0380 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.155+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(626..626 src has 1..626) v4 ==== 5793+0+0 (unknown 1853797482 0 0) 0x7f4a08043ff0 con 0x7f4a200a0380 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.155+0000 7f4a16ffd700 10 client.134232.objecter ms_dispatch 0x7f4a20000fd0 osd_map(626..626 src has 1..626) v4 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.155+0000 7f4a16ffd700 3 client.134232.objecter handle_osd_map got epochs [626,626] > 0 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.155+0000 7f4a16ffd700 3 client.134232.objecter handle_osd_map decoding full epoch 626 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.155+0000 7f4a16ffd700 20 client.134232.objecter dump_active .. 0 homeless 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.155+0000 7f4a1e59c700 10 monclient: get_auth_request con 0x7f4a00038750 auth_method 0 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.155+0000 7f4a16ffd700 10 client.134232.objecter ms_handle_connect 0x7f4a00038750 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.159+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f4a08014a30 con 0x7f4a200a0380 2024-09-18T12:00:01.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.159+0000 7f4a16ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.159+0000 7f4a16ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:01.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.387+0000 7f4a1f59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:00:01.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.387+0000 7f4a1f59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:00:01.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.387+0000 7f4a1f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.387+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4a2005f2e0 con 0x7f4a200a0380 2024-09-18T12:00:01.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.387+0000 7f4a16ffd700 1 -- 172.21.15.112:0/3729723972 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1099710963) 0x7f4a08014430 con 0x7f4a200a0380 2024-09-18T12:00:01.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.387+0000 7f4a16ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:00:01.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.387+0000 7f4a16ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:01.391 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:00:01.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 10 librados: watch_flush enter 2024-09-18T12:00:01.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 10 librados: watch_flush exit 2024-09-18T12:00:01.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 20 client.134232.objecter shutdown clearing up homeless session... 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 10 client.134232.objecter successfully canceled tick 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 >> v1:172.21.15.129:6812/20806 conn(0x7f4a00038750 legacy=0x7f4a0003ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 10 monclient: shutdown 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 >> v1:172.21.15.112:6789/0 conn(0x7f4a200a0380 legacy=0x7f4a20146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 shutdown_connections 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 >> 172.21.15.112:0/3729723972 conn(0x7f4a20068100 msgr2=0x7f4a20135390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 shutdown_connections 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 1 -- 172.21.15.112:0/3729723972 wait complete. 2024-09-18T12:00:01.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.391+0000 7f4a1f59e700 1 librados: shutdown 2024-09-18T12:00:01.404 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:00:01.404 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:00:01.404 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:00:01.404 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:00:01.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient: get_monmap_and_config 2024-09-18T12:00:01.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient: build_initial_monmap 2024-09-18T12:00:01.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient: monmap: 2024-09-18T12:00:01.514 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:01.514 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.514 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:01.514305+0000 2024-09-18T12:00:01.514 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:01.514305+0000 2024-09-18T12:00:01.514 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:01.515 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.515 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:01.515 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:01.515 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:01.515 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 1 Processor -- start 2024-09-18T12:00:01.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 1 -- start start 2024-09-18T12:00:01.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient: init 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f032c0a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f032c0a0750 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f032c1385b0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c097270 con 0x7f032c1385b0 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c0a3170 con 0x7f032c0a0750 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c096cb0 con 0x7f032c0a0380 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032b59e700 10 monclient(hunting): authenticate will time out at 1773.355439s 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032ad9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f032c1385b0 0x7f032c13a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39742/0 (socket says 172.21.15.112:39742) 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032a59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f032c0a0380 0x7f032c097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59572/0 (socket says 172.21.15.112:59572) 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.511+0000 7f032ad9d700 1 -- 172.21.15.112:0/402515824 learned_addr learned my addr 172.21.15.112:0/402515824 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032ad9d700 1 -- 172.21.15.112:0/402515824 >> v1:172.21.15.112:6789/0 conn(0x7f032c1385b0 legacy=0x7f032c13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032a59c700 1 -- 172.21.15.112:0/402515824 >> v1:172.21.15.112:6790/0 conn(0x7f032c0a0380 legacy=0x7f032c097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f0329d9b700 1 -- 172.21.15.112:0/402515824 >> v1:172.21.15.129:6789/0 conn(0x7f032c0a0750 legacy=0x7f032c0a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365967238 0 0) 0x7f032c097270 con 0x7f032c1385b0 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient(hunting): my global_id is 134235 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030c003570 con 0x7f032c1385b0 2024-09-18T12:00:01.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1709502319 0 0) 0x7f030c003570 con 0x7f032c1385b0 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 >> v1:172.21.15.112:6790/0 conn(0x7f032c0a0380 legacy=0x7f032c097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 >> v1:172.21.15.129:6789/0 conn(0x7f032c0a0750 legacy=0x7f032c0a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient: _finish_hunting 0 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 1 monclient: found mon.noname-a 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c0a1420 con 0x7f032c1385b0 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient: _finish_auth 0 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032b59e700 5 monclient: authenticate success, global_id 134235 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0324001940 con 0x7f032c1385b0 2024-09-18T12:00:01.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032959a700 10 monclient: dump: 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.522 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.523 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.515+0000 7f032b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:01.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0324001c10 con 0x7f032c1385b0 2024-09-18T12:00:01.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:01.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032959a700 1 -- 172.21.15.112:0/402515824 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0324004a90 con 0x7f032c1385b0 2024-09-18T12:00:01.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:01.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032959a700 10 monclient: dump: 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 4 set_mon_vals no callback set 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 10 monclient: shutdown 2024-09-18T12:00:01.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 1 -- 172.21.15.112:0/402515824 >> v1:172.21.15.112:6789/0 conn(0x7f032c1385b0 legacy=0x7f032c13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 1 -- 172.21.15.112:0/402515824 shutdown_connections 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 1 -- 172.21.15.112:0/402515824 >> 172.21.15.112:0/402515824 conn(0x7f032c0a5990 msgr2=0x7f032c0a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 1 -- 172.21.15.112:0/402515824 shutdown_connections 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 1 -- 172.21.15.112:0/402515824 wait complete. 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 10 monclient: build_initial_monmap 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 10 monclient: monmap: 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:01.527 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 1 librados: starting msgr at 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.519+0000 7f032b59e700 1 librados: starting objecter 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 1 Processor -- start 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 1 -- start start 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 1 librados: setting wanted keys 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 1 librados: calling monclient init 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 10 monclient: init 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:01.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f032c0a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f032c0a0750 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f032c1385b0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c1389f0 con 0x7f032c1385b0 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c141ca0 con 0x7f032c0a0750 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c137e90 con 0x7f032c0a0380 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032b59e700 10 monclient(hunting): authenticate will time out at 1773.364940s 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032a59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f032c0a0380 0x7f032c141590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59578/0 (socket says 172.21.15.112:59578) 2024-09-18T12:00:01.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032a59c700 1 -- 172.21.15.112:0/748370898 learned_addr learned my addr 172.21.15.112:0/748370898 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:01.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032a59c700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6790/0 conn(0x7f032c0a0380 legacy=0x7f032c141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 client.?.objecter ms_handle_connect 0x7f032c0a0380 2024-09-18T12:00:01.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 client.?.objecter ms_handle_connect 0x7f032c1385b0 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f032ad9d700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6789/0 conn(0x7f032c1385b0 legacy=0x7f032c149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 client.?.objecter ms_handle_connect 0x7f032c0a0750 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f0329d9b700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.129:6789/0 conn(0x7f032c0a0750 legacy=0x7f032c09dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385422797 0 0) 0x7f032c137e90 con 0x7f032c0a0380 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): my global_id is 104534 2024-09-18T12:00:01.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310004f10 con 0x7f032c0a0380 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413428096 0 0) 0x7f032c1389f0 con 0x7f032c1385b0 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): my global_id is 134238 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c137e90 con 0x7f032c1385b0 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257012404 0 0) 0x7f032c141ca0 con 0x7f032c0a0750 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 10 monclient(hunting): my global_id is 144127 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.523+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c1389f0 con 0x7f032c0a0750 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 567659140 0 0) 0x7f032c137e90 con 0x7f032c1385b0 2024-09-18T12:00:01.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f032c141ca0 con 0x7f032c1385b0 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0324004830 con 0x7f032c1385b0 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): dump: 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.538 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3158119864 0 0) 0x7f0310004f10 con 0x7f032c0a0380 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f032c137e90 con 0x7f032c0a0380 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f031c001c50 con 0x7f032c0a0380 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): dump: 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.539 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157654570 0 0) 0x7f032c1389f0 con 0x7f032c0a0750 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0310004f10 con 0x7f032c0a0750 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0314001f00 con 0x7f032c0a0750 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient(hunting): dump: 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.540 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.541 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.541 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.541 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.541 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.541 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.541 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1774483133 0 0) 0x7f032c141ca0 con 0x7f032c1385b0 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6790/0 conn(0x7f032c0a0380 legacy=0x7f032c141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.129:6789/0 conn(0x7f032c0a0750 legacy=0x7f032c09dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient: _finish_hunting 0 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 monclient: found mon.a 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c0981b0 con 0x7f032c1385b0 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient: _finish_auth 0 2024-09-18T12:00:01.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 5 monclient: authenticate success, global_id 134238 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 10 monclient: _renew_subs 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c098490 con 0x7f032c1385b0 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 10 client.134238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 10 monclient: _renew_subs 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c09e5d0 con 0x7f032c1385b0 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 1 librados: init done 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0324001500 con 0x7f032c1385b0 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0324004ee0 con 0x7f032c1385b0 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f0323fff700 4 set_mon_vals no callback set 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 monclient: dump: 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f03240050c0 con 0x7f032c1385b0 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0304006590 con 0x7f0304002840 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(627..627 src has 1..627) v4 ==== 5806+0+0 (unknown 3594278032 0 0) 0x7f03240440f0 con 0x7f032c1385b0 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 10 client.134238.objecter ms_dispatch 0x7f032c000fd0 osd_map(627..627 src has 1..627) v4 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 3 client.134238.objecter handle_osd_map got epochs [627,627] > 0 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 3 client.134238.objecter handle_osd_map decoding full epoch 627 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f03227fc700 20 client.134238.objecter dump_active .. 0 homeless 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.527+0000 7f0329d9b700 10 monclient: get_auth_request con 0x7f03100386b0 auth_method 0 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 client.134238.objecter ms_handle_connect 0x7f0304002840 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 client.134238.objecter resend_mon_ops 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f032a59c700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6789/0 conn(0x7f0304002840 legacy=0x7f0304004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871071662 0 0) 0x7f0304006590 con 0x7f0304002840 2024-09-18T12:00:01.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 monclient(hunting): my global_id is 134241 2024-09-18T12:00:01.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c141ca0 con 0x7f0304002840 2024-09-18T12:00:01.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 110362443 0 0) 0x7f032c141ca0 con 0x7f0304002840 2024-09-18T12:00:01.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0304006590 con 0x7f0304002840 2024-09-18T12:00:01.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f031c004fb0 con 0x7f0304002840 2024-09-18T12:00:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 monclient: dump: 2024-09-18T12:00:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.548 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.548 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.548 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.548 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.548 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.548 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.549 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 client.134238.objecter ms_handle_connect 0x7f03100386b0 2024-09-18T12:00:01.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3189739049 0 0) 0x7f0304006590 con 0x7f0304002840 2024-09-18T12:00:01.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f030400a140 con 0x7f0304002840 2024-09-18T12:00:01.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f031c0051c0 con 0x7f0304002840 2024-09-18T12:00:01.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:01.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6789/0 conn(0x7f0304002840 legacy=0x7f0304004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.531+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6789/0 conn(0x7f0304002840 legacy=0x7f0304004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:01.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.543+0000 7f032b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:00:01.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.543+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0304009f10 con 0x7f0304006900 2024-09-18T12:00:01.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 client.134238.objecter ms_handle_connect 0x7f0304006900 2024-09-18T12:00:01.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 client.134238.objecter resend_mon_ops 2024-09-18T12:00:01.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032a59c700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6789/0 conn(0x7f0304006900 legacy=0x7f0304006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42481811 0 0) 0x7f0304009f10 con 0x7f0304006900 2024-09-18T12:00:01.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 monclient(hunting): my global_id is 134244 2024-09-18T12:00:01.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304006590 con 0x7f0304006900 2024-09-18T12:00:01.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2365340779 0 0) 0x7f0304006590 con 0x7f0304006900 2024-09-18T12:00:01.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0304009f10 con 0x7f0304006900 2024-09-18T12:00:01.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f031c0056c0 con 0x7f0304006900 2024-09-18T12:00:01.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:01.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 monclient: dump: 2024-09-18T12:00:01.556 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.556 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.556 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.556 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.557 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.557 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.557 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.557 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.557 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.557 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 668834595 0 0) 0x7f0304009f10 con 0x7f0304006900 2024-09-18T12:00:01.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0304008760 con 0x7f0304006900 2024-09-18T12:00:01.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4003 (unknown 0 0 4149437995) 0x7f031c005a90 con 0x7f0304006900 2024-09-18T12:00:01.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:00:01.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:01.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6789/0 conn(0x7f0304006900 legacy=0x7f0304006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f03227fc700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6789/0 conn(0x7f0304006900 legacy=0x7f0304006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:01.559 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:00:01.559 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:00:01.559 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:00:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T12:00:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 70, 2024-09-18T12:00:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:00:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:00:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:00:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:00:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T12:00:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:00:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:00:01.562 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.562 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.562 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.562 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.562 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.562 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.563 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.563 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.563 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.563 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.563 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:00:01.563 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:00:01.564 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.564 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.564 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.564 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.564 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.565 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.565 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.565 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.565 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:00:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:00:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:00:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:00:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:00:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:00:01.566 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:00:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:00:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:00:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:00:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:00:01.567 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:00:01.568 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:00:01.568 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:01.568 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:00:01.568 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.568 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.568 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.568 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.569 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.569 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.569 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.569 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.569 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.569 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.569 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:00:01.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.570 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:00:01.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.570 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:00:01.570 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:00:01.570 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.571 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.571 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.571 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:00:01.571 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.571 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.572 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:00:01.572 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:00:01.572 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.572 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.572 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.573 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:00:01.573 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:00:01.573 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.573 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.573 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.573 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:00:01.574 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.574 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.574 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:00:01.574 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:00:01.574 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.574 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.575 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.575 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:00:01.575 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:00:01.575 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.575 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.576 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.576 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:00:01.576 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.576 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.576 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:00:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.578 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:00:01.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.579 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.579 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.579 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T12:00:01.579 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.579 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.580 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:00:01.580 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:00:01.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 10 librados: watch_flush enter 2024-09-18T12:00:01.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 10 librados: watch_flush exit 2024-09-18T12:00:01.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 20 client.134238.objecter shutdown clearing up homeless session... 2024-09-18T12:00:01.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 10 client.134238.objecter successfully canceled tick 2024-09-18T12:00:01.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.129:6812/20806 conn(0x7f03100386b0 legacy=0x7f031003ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 10 monclient: shutdown 2024-09-18T12:00:01.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:01.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 >> v1:172.21.15.112:6789/0 conn(0x7f032c1385b0 legacy=0x7f032c149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032ad9d700 1 -- 172.21.15.112:0/748370898 reap_dead start 2024-09-18T12:00:01.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032ad9d700 1 -- 172.21.15.112:0/748370898 reap_dead start 2024-09-18T12:00:01.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 shutdown_connections 2024-09-18T12:00:01.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 >> 172.21.15.112:0/748370898 conn(0x7f032c0a5990 msgr2=0x7f032c138c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:01.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 shutdown_connections 2024-09-18T12:00:01.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 1 -- 172.21.15.112:0/748370898 wait complete. 2024-09-18T12:00:01.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.547+0000 7f032b59e700 1 librados: shutdown 2024-09-18T12:00:01.584 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:00:01.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:00:01.585 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:00:01.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.695+0000 7fab7917f700 10 monclient: get_monmap_and_config 2024-09-18T12:00:01.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.695+0000 7fab7917f700 10 monclient: build_initial_monmap 2024-09-18T12:00:01.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.695+0000 7fab7917f700 10 monclient: monmap: 2024-09-18T12:00:01.698 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:01.698 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.698 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:01.698767+0000 2024-09-18T12:00:01.698 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:01.698767+0000 2024-09-18T12:00:01.699 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:01.699 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.699 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:01.699 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:01.699 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:01.699 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.695+0000 7fab7917f700 1 Processor -- start 2024-09-18T12:00:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.695+0000 7fab7917f700 1 -- start start 2024-09-18T12:00:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.695+0000 7fab7917f700 10 monclient: init 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 10 monclient(hunting): picked mon.noname-c con 0x7fab740a0380 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 10 monclient(hunting): picked mon.noname-b con 0x7fab740a0e60 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 10 monclient(hunting): picked mon.noname-a con 0x7fab74097970 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab74097550 con 0x7fab74097970 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab74096cb0 con 0x7fab740a0380 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab740a1440 con 0x7fab740a0e60 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 10 monclient(hunting): authenticate will time out at 1773.540599s 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab737fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fab740a0e60 0x7fab74137390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59582/0 (socket says 172.21.15.112:59582) 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab737fe700 1 -- 172.21.15.112:0/3006107172 learned_addr learned my addr 172.21.15.112:0/3006107172 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7897e700 1 -- 172.21.15.112:0/3006107172 >> v1:172.21.15.112:6789/0 conn(0x7fab74097970 legacy=0x7fab7413aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab737fe700 1 -- 172.21.15.112:0/3006107172 >> v1:172.21.15.112:6790/0 conn(0x7fab740a0e60 legacy=0x7fab74137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab73fff700 1 -- 172.21.15.112:0/3006107172 >> v1:172.21.15.129:6789/0 conn(0x7fab740a0380 legacy=0x7fab740a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275768550 0 0) 0x7fab74097550 con 0x7fab74097970 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): my global_id is 134247 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab5c003570 con 0x7fab74097970 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554162154 0 0) 0x7fab740a1440 con 0x7fab740a0e60 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): my global_id is 104537 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab74097550 con 0x7fab740a0e60 2024-09-18T12:00:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131168458 0 0) 0x7fab74096cb0 con 0x7fab740a0380 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient(hunting): my global_id is 144130 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab740a1440 con 0x7fab740a0380 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2496277246 0 0) 0x7fab74097550 con 0x7fab740a0e60 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 >> v1:172.21.15.129:6789/0 conn(0x7fab740a0380 legacy=0x7fab740a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 >> v1:172.21.15.112:6789/0 conn(0x7fab74097970 legacy=0x7fab7413aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 monclient: found mon.noname-b 2024-09-18T12:00:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab740a1de0 con 0x7fab740a0e60 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 10 monclient: _finish_auth 0 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab7917f700 5 monclient: authenticate success, global_id 104537 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.699+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fab680019e0 con 0x7fab740a0e60 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 10 monclient: dump: 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab68001bc0 con 0x7fab740a0e60 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 1 -- 172.21.15.112:0/3006107172 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fab68004a90 con 0x7fab740a0e60 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 4 set_mon_vals no callback set 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab72ffd700 10 monclient: dump: 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient: shutdown 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- 172.21.15.112:0/3006107172 >> v1:172.21.15.112:6790/0 conn(0x7fab740a0e60 legacy=0x7fab74137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- 172.21.15.112:0/3006107172 shutdown_connections 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- 172.21.15.112:0/3006107172 >> 172.21.15.112:0/3006107172 conn(0x7fab74068100 msgr2=0x7fab740a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- 172.21.15.112:0/3006107172 shutdown_connections 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- 172.21.15.112:0/3006107172 wait complete. 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient: build_initial_monmap 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient: monmap: 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 librados: starting msgr at 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 librados: starting objecter 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 Processor -- start 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- start start 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 librados: setting wanted keys 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 librados: calling monclient init 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient: init 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient(hunting): picked mon.noname-c con 0x7fab74097970 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:01.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient(hunting): picked mon.noname-a con 0x7fab740a0380 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient(hunting): picked mon.noname-b con 0x7fab740a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab741227d0 con 0x7fab740a0380 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7413db60 con 0x7fab740a0e60 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab74120260 con 0x7fab74097970 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab7917f700 10 monclient(hunting): authenticate will time out at 1773.547009s 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab73fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fab74097970 0x7fab7413d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59594/0 (socket says 172.21.15.112:59594) 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.703+0000 7fab73fff700 1 -- 172.21.15.112:0/3593568343 learned_addr learned my addr 172.21.15.112:0/3593568343 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab73fff700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6790/0 conn(0x7fab74097970 legacy=0x7fab7413d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 client.?.objecter ms_handle_connect 0x7fab74097970 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 client.?.objecter ms_handle_connect 0x7fab740a0380 2024-09-18T12:00:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab737fe700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6789/0 conn(0x7fab740a0380 legacy=0x7fab741466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 client.?.objecter ms_handle_connect 0x7fab740a0e60 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7897e700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.129:6789/0 conn(0x7fab740a0e60 legacy=0x7fab74149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920582273 0 0) 0x7fab741227d0 con 0x7fab740a0380 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): my global_id is 134250 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab60004f10 con 0x7fab740a0380 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340778540 0 0) 0x7fab74120260 con 0x7fab74097970 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): my global_id is 104540 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab741227d0 con 0x7fab74097970 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820637409 0 0) 0x7fab7413db60 con 0x7fab740a0e60 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): my global_id is 144133 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab74120260 con 0x7fab740a0e60 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3765721580 0 0) 0x7fab60004f10 con 0x7fab740a0380 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab7413db60 con 0x7fab740a0380 2024-09-18T12:00:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fab68004830 con 0x7fab740a0380 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): dump: 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 113861121 0 0) 0x7fab741227d0 con 0x7fab74097970 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab60004f10 con 0x7fab74097970 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fab64001cf0 con 0x7fab74097970 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): dump: 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3953897016 0 0) 0x7fab74120260 con 0x7fab740a0e60 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab741227d0 con 0x7fab740a0e60 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fab6c001fe0 con 0x7fab740a0e60 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient(hunting): dump: 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446032058 0 0) 0x7fab60004f10 con 0x7fab74097970 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.129:6789/0 conn(0x7fab740a0e60 legacy=0x7fab74149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6789/0 conn(0x7fab740a0380 legacy=0x7fab741466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient: _finish_hunting 0 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 monclient: found mon.c 2024-09-18T12:00:01.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab741224f0 con 0x7fab74097970 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient: _finish_auth 0 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 5 monclient: authenticate success, global_id 104540 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 10 monclient: _renew_subs 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab74122ab0 con 0x7fab74097970 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 10 client.104540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 10 monclient: _renew_subs 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7413e950 con 0x7fab74097970 2024-09-18T12:00:01.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab7917f700 1 librados: init done 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab640022b0 con 0x7fab74097970 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fab64002600 con 0x7fab74097970 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab727fc700 4 set_mon_vals no callback set 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.707+0000 7fab5bfff700 10 monclient: dump: 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.736 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab7917f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fab7405f2e0 con 0x7fab74121620 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fab64002890 con 0x7fab74097970 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(627..627 src has 1..627) v4 ==== 5806+0+0 (unknown 3594278032 0 0) 0x7fab64043fd0 con 0x7fab74097970 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 client.104540.objecter ms_dispatch 0x7fab74000fd0 osd_map(627..627 src has 1..627) v4 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 3 client.104540.objecter handle_osd_map got epochs [627,627] > 0 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 3 client.104540.objecter handle_osd_map decoding full epoch 627 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 20 client.104540.objecter dump_active .. 0 homeless 2024-09-18T12:00:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab7897e700 10 monclient: get_auth_request con 0x7fab600385e0 auth_method 0 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab737fe700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6790/0 conn(0x7fab74121620 legacy=0x7fab7405e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 client.104540.objecter ms_handle_connect 0x7fab74121620 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 client.104540.objecter resend_mon_ops 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470057398 0 0) 0x7fab7405f2e0 con 0x7fab74121620 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 monclient(hunting): my global_id is 104543 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab60004f10 con 0x7fab74121620 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3751060533 0 0) 0x7fab60004f10 con 0x7fab74121620 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab7405f2e0 con 0x7fab74121620 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fab680052e0 con 0x7fab74121620 2024-09-18T12:00:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 10 monclient: dump: 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 515273438 0 0) 0x7fab7405f2e0 con 0x7fab74121620 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.711+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab740008d0 con 0x7fab74121620 2024-09-18T12:00:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.715+0000 7fab5bfff700 10 client.104540.objecter ms_handle_connect 0x7fab600385e0 2024-09-18T12:00:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.715+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fab68005440 con 0x7fab74121620 2024-09-18T12:00:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.715+0000 7fab5bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.715+0000 7fab5bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.715+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6790/0 conn(0x7fab74121620 legacy=0x7fab7405e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.715+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6790/0 conn(0x7fab74121620 legacy=0x7fab7405e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:01.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.723+0000 7fab7917f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.723+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fab7413e1a0 con 0x7fab7413dda0 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.723+0000 7fab737fe700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6790/0 conn(0x7fab7413dda0 legacy=0x7fab7409d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.723+0000 7fab5bfff700 10 client.104540.objecter ms_handle_connect 0x7fab7413dda0 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.723+0000 7fab5bfff700 10 client.104540.objecter resend_mon_ops 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.723+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020964977 0 0) 0x7fab7413e1a0 con 0x7fab7413dda0 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 10 monclient(hunting): my global_id is 104546 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab7405f2e0 con 0x7fab7413dda0 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1036354724 0 0) 0x7fab7405f2e0 con 0x7fab7413dda0 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab7413e1a0 con 0x7fab7413dda0 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fab68005b20 con 0x7fab7413dda0 2024-09-18T12:00:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 10 monclient: dump: 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2497972740 0 0) 0x7fab7413e1a0 con 0x7fab7413dda0 2024-09-18T12:00:01.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fab7409db80 con 0x7fab7413dda0 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 381487575) 0x7fab68007060 con 0x7fab7413dda0 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6790/0 conn(0x7fab7413dda0 legacy=0x7fab7409d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab5bfff700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6790/0 conn(0x7fab7413dda0 legacy=0x7fab7409d470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:00:01.744 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 70, 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.745 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:00:01.747 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.748 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:00:01.749 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.750 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:00:01.751 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:00:01.752 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.753 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.754 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.754 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.754 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:00:01.754 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:00:01.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.755 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.756 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.756 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:01.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.756 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.756 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:00:01.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.756 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.756 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.757 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.758 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.758 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:01.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.758 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.758 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:00:01.758 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 10 librados: watch_flush enter 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 10 librados: watch_flush exit 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 20 client.104540.objecter shutdown clearing up homeless session... 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 10 client.104540.objecter successfully canceled tick 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.129:6812/20806 conn(0x7fab600385e0 legacy=0x7fab6003aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 10 monclient: shutdown 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 >> v1:172.21.15.112:6790/0 conn(0x7fab74097970 legacy=0x7fab7413d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7897e700 1 -- 172.21.15.112:0/3593568343 reap_dead start 2024-09-18T12:00:01.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7897e700 1 -- 172.21.15.112:0/3593568343 reap_dead start 2024-09-18T12:00:01.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 shutdown_connections 2024-09-18T12:00:01.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 >> 172.21.15.112:0/3593568343 conn(0x7fab74068100 msgr2=0x7fab740a4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:01.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 shutdown_connections 2024-09-18T12:00:01.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 1 -- 172.21.15.112:0/3593568343 wait complete. 2024-09-18T12:00:01.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.727+0000 7fab7917f700 1 librados: shutdown 2024-09-18T12:00:01.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:00:01.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:00:01.761 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:00:01.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient: get_monmap_and_config 2024-09-18T12:00:01.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient: build_initial_monmap 2024-09-18T12:00:01.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient: monmap: 2024-09-18T12:00:01.878 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:01.878 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.879 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:01.877060+0000 2024-09-18T12:00:01.879 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:01.877060+0000 2024-09-18T12:00:01.879 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:01.879 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.879 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:01.879 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:01.880 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:01.880 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 1 Processor -- start 2024-09-18T12:00:01.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 1 -- start start 2024-09-18T12:00:01.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient: init 2024-09-18T12:00:01.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:01.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:00:01.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient(hunting): picked mon.noname-b con 0x7f19800a2040 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:01.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient(hunting): picked mon.noname-a con 0x7f198012edc0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:01.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient(hunting): picked mon.noname-c con 0x7f198012f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:01.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800981a0 con 0x7f198012edc0 2024-09-18T12:00:01.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f198006c120 con 0x7f198012f8a0 2024-09-18T12:00:01.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f198006eef0 con 0x7f19800a2040 2024-09-18T12:00:01.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:01.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19866f3700 10 monclient(hunting): authenticate will time out at 1773.718003s 2024-09-18T12:00:01.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19856f1700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f19800a2040 0x7f1980097a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59628/0 (socket says 172.21.15.112:59628) 2024-09-18T12:00:01.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f1984ef0700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f198012edc0 0x7f198012f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39772/0 (socket says 172.21.15.112:39772) 2024-09-18T12:00:01.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.875+0000 7f19856f1700 1 -- 172.21.15.112:0/899361868 learned_addr learned my addr 172.21.15.112:0/899361868 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:01.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1984ef0700 1 -- 172.21.15.112:0/899361868 >> v1:172.21.15.112:6789/0 conn(0x7f198012edc0 legacy=0x7f198012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19856f1700 1 -- 172.21.15.112:0/899361868 >> v1:172.21.15.112:6790/0 conn(0x7f19800a2040 legacy=0x7f1980097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1985ef2700 1 -- 172.21.15.112:0/899361868 >> v1:172.21.15.129:6789/0 conn(0x7f198012f8a0 legacy=0x7f1980137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350579988 0 0) 0x7f19800981a0 con 0x7f198012edc0 2024-09-18T12:00:01.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): my global_id is 134253 2024-09-18T12:00:01.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f196c003570 con 0x7f198012edc0 2024-09-18T12:00:01.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208261828 0 0) 0x7f198006eef0 con 0x7f19800a2040 2024-09-18T12:00:01.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): my global_id is 104549 2024-09-18T12:00:01.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19800981a0 con 0x7f19800a2040 2024-09-18T12:00:01.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307485544 0 0) 0x7f198006c120 con 0x7f198012f8a0 2024-09-18T12:00:01.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient(hunting): my global_id is 144136 2024-09-18T12:00:01.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f198006eef0 con 0x7f198012f8a0 2024-09-18T12:00:01.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688938448 0 0) 0x7f196c003570 con 0x7f198012edc0 2024-09-18T12:00:01.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 >> v1:172.21.15.112:6790/0 conn(0x7f19800a2040 legacy=0x7f1980097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 >> v1:172.21.15.129:6789/0 conn(0x7f198012f8a0 legacy=0x7f1980137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: _finish_hunting 0 2024-09-18T12:00:01.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 monclient: found mon.noname-a 2024-09-18T12:00:01.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:01.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:00:01.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1980096dd0 con 0x7f198012edc0 2024-09-18T12:00:01.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: _finish_auth 0 2024-09-18T12:00:01.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:01.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f197c001cc0 con 0x7f198012edc0 2024-09-18T12:00:01.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:01.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: dump: 2024-09-18T12:00:01.900 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 5 monclient: authenticate success, global_id 134253 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f197c001f50 con 0x7f198012edc0 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:01.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 4 set_mon_vals no callback set 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 1 -- 172.21.15.112:0/899361868 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f197c004d50 con 0x7f198012edc0 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f1977fff700 10 monclient: dump: 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.902 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 10 monclient: shutdown 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 1 -- 172.21.15.112:0/899361868 >> v1:172.21.15.112:6789/0 conn(0x7f198012edc0 legacy=0x7f198012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 1 -- 172.21.15.112:0/899361868 shutdown_connections 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 1 -- 172.21.15.112:0/899361868 >> 172.21.15.112:0/899361868 conn(0x7f1980067da0 msgr2=0x7f19800a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 1 -- 172.21.15.112:0/899361868 shutdown_connections 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.879+0000 7f19866f3700 1 -- 172.21.15.112:0/899361868 wait complete. 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient: build_initial_monmap 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient: monmap: 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:01.903 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 librados: starting msgr at 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 librados: starting objecter 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 Processor -- start 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 -- start start 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 librados: setting wanted keys 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 librados: calling monclient init 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient: init 2024-09-18T12:00:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient(hunting): picked mon.noname-c con 0x7f19800a2040 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient(hunting): picked mon.noname-a con 0x7f198012edc0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient(hunting): picked mon.noname-b con 0x7f198012f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1980134b30 con 0x7f198012edc0 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f198013f070 con 0x7f198012f8a0 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f198013e8a0 con 0x7f19800a2040 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19866f3700 10 monclient(hunting): authenticate will time out at 1773.726611s 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19856f1700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f19800a2040 0x7f198013e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:59634/0 (socket says 172.21.15.112:59634) 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.883+0000 7f19856f1700 1 -- 172.21.15.112:0/610876693 learned_addr learned my addr 172.21.15.112:0/610876693 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19856f1700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.112:6790/0 conn(0x7f19800a2040 legacy=0x7f198013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 client.?.objecter ms_handle_connect 0x7f19800a2040 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 client.?.objecter ms_handle_connect 0x7f198012edc0 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f1984ef0700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.112:6789/0 conn(0x7f198012edc0 legacy=0x7f1980137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286704556 0 0) 0x7f198013e8a0 con 0x7f19800a2040 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): my global_id is 104552 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1964004f10 con 0x7f19800a2040 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 client.?.objecter ms_handle_connect 0x7f198012f8a0 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f1985ef2700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6789/0 conn(0x7f198012f8a0 legacy=0x7f19801464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250226009 0 0) 0x7f1980134b30 con 0x7f198012edc0 2024-09-18T12:00:01.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): my global_id is 134256 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f198013e8a0 con 0x7f198012edc0 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1755176319 0 0) 0x7f1964004f10 con 0x7f19800a2040 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1980134b30 con 0x7f19800a2040 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1970001cb0 con 0x7f19800a2040 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): dump: 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.907 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357458995 0 0) 0x7f198006eef0 con 0x7f198012f8a0 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): my global_id is 144139 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1964004f10 con 0x7f198012f8a0 2024-09-18T12:00:01.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 598244516 0 0) 0x7f198013e8a0 con 0x7f198012edc0 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f198006eef0 con 0x7f198012edc0 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f197c0016e0 con 0x7f198012edc0 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient(hunting): dump: 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.909 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3124690682 0 0) 0x7f1980134b30 con 0x7f19800a2040 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6789/0 conn(0x7f198012f8a0 legacy=0x7f19801464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.112:6789/0 conn(0x7f198012edc0 legacy=0x7f1980137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient: _finish_hunting 0 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 monclient: found mon.c 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19800a0480 con 0x7f19800a2040 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 10 monclient: _finish_auth 0 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19757fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 5 monclient: authenticate success, global_id 104552 2024-09-18T12:00:01.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 10 monclient: _renew_subs 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1980134620 con 0x7f19800a2040 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 10 client.104552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 10 monclient: _renew_subs 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1980135180 con 0x7f19800a2040 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 1 librados: init done 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.887+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f19801367d0 con 0x7f1980138550 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1970004be0 con 0x7f19800a2040 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1970004f30 con 0x7f19800a2040 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient: dump: 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f1970005110 con 0x7f19800a2040 2024-09-18T12:00:01.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19777fe700 4 set_mon_vals no callback set 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(627..627 src has 1..627) v4 ==== 5806+0+0 (unknown 3594278032 0 0) 0x7f1970044180 con 0x7f19800a2040 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 client.104552.objecter ms_dispatch 0x7f1980000fd0 osd_map(627..627 src has 1..627) v4 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 3 client.104552.objecter handle_osd_map got epochs [627,627] > 0 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 3 client.104552.objecter handle_osd_map decoding full epoch 627 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 20 client.104552.objecter dump_active .. 0 homeless 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f1985ef2700 10 monclient: get_auth_request con 0x7f196403a820 auth_method 0 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 client.104552.objecter ms_handle_connect 0x7f1980138550 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 client.104552.objecter resend_mon_ops 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f1984ef0700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6789/0 conn(0x7f1980138550 legacy=0x7f198009bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158705737 0 0) 0x7f19801367d0 con 0x7f1980138550 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient(hunting): my global_id is 144142 2024-09-18T12:00:01.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1980134b30 con 0x7f1980138550 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1787233102 0 0) 0x7f1980134b30 con 0x7f1980138550 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19801367d0 con 0x7f1980138550 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f197c001940 con 0x7f1980138550 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 monclient: dump: 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.914 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 293808881 0 0) 0x7f19801367d0 con 0x7f1980138550 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f198005f2e0 con 0x7f1980138550 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.891+0000 7f19757fa700 10 client.104552.objecter ms_handle_connect 0x7f196403a820 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.895+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f197c007580 con 0x7f1980138550 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.895+0000 7f19757fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.895+0000 7f19757fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.895+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6789/0 conn(0x7f1980138550 legacy=0x7f198009bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.895+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6789/0 conn(0x7f1980138550 legacy=0x7f198009bb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:01.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.903+0000 7f19866f3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:00:01.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.903+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1980063680 con 0x7f198002ad10 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f1984ef0700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6789/0 conn(0x7f198002ad10 legacy=0x7f198009c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 client.104552.objecter ms_handle_connect 0x7f198002ad10 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 client.104552.objecter resend_mon_ops 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877841347 0 0) 0x7f1980063680 con 0x7f198002ad10 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 monclient(hunting): my global_id is 144145 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19801367d0 con 0x7f198002ad10 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1045356448 0 0) 0x7f19801367d0 con 0x7f198002ad10 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1980063680 con 0x7f198002ad10 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f197c005a20 con 0x7f198002ad10 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:01.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 monclient: dump: 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1472346726 0 0) 0x7f1980063680 con 0x7f198002ad10 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f198009c940 con 0x7f198002ad10 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 4212002698) 0x7f197c007830 con 0x7f198002ad10 2024-09-18T12:00:01.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6789/0 conn(0x7f198002ad10 legacy=0x7f198009c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19757fa700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6789/0 conn(0x7f198002ad10 legacy=0x7f198009c230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 70, 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:00:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 3, 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.921 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:00:01.922 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:01.923 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.924 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:00:01.925 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.926 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.928 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:00:01.929 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:00:01.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19866f3700 10 librados: watch_flush enter 2024-09-18T12:00:01.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19866f3700 10 librados: watch_flush exit 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19866f3700 20 client.104552.objecter shutdown clearing up homeless session... 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.907+0000 7f19866f3700 10 client.104552.objecter successfully canceled tick 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.129:6812/20806 conn(0x7f196403a820 legacy=0x7f196403ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 10 monclient: shutdown 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 >> v1:172.21.15.112:6790/0 conn(0x7f19800a2040 legacy=0x7f198013e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f1985ef2700 1 -- 172.21.15.112:0/610876693 reap_dead start 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f1985ef2700 1 -- 172.21.15.112:0/610876693 reap_dead start 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 shutdown_connections 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 >> 172.21.15.112:0/610876693 conn(0x7f1980067da0 msgr2=0x7f198006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 shutdown_connections 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 1 -- 172.21.15.112:0/610876693 wait complete. 2024-09-18T12:00:01.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:01.911+0000 7f19866f3700 1 librados: shutdown 2024-09-18T12:00:01.933 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:00:01.933 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:00:01.933 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:00:01.933 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:00:01.933 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:00:01.933 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:00:01.934 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:00:01.951 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:00:01.952 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:00:01.952 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:00:02.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient: get_monmap_and_config 2024-09-18T12:00:02.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient: build_initial_monmap 2024-09-18T12:00:02.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient: monmap: 2024-09-18T12:00:02.055 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:02.055 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:02.056 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:02.056707+0000 2024-09-18T12:00:02.056 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:02.056707+0000 2024-09-18T12:00:02.056 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:02.056 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:02.056 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:02.056 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:02.056 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:02.056 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 1 Processor -- start 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 1 -- start start 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient: init 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient(hunting): picked mon.noname-c con 0x7fb328067680 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient(hunting): picked mon.noname-b con 0x7fb328097a20 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient(hunting): picked mon.noname-a con 0x7fb3281387a0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb328067a50 con 0x7fb3281387a0 2024-09-18T12:00:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280a1390 con 0x7fb328067680 2024-09-18T12:00:02.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280a2070 con 0x7fb328097a20 2024-09-18T12:00:02.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:02.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb32de07700 10 monclient(hunting): authenticate will time out at 1773.897570s 2024-09-18T12:00:02.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb31ffff700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fb328097a20 legacy=0x7fb3280a0c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:02.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb327fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb3281387a0 0x7fb32813ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39812/0 (socket says 172.21.15.112:39812) 2024-09-18T12:00:02.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.055+0000 7fb327fff700 1 -- 172.21.15.112:0/253233713 learned_addr learned my addr 172.21.15.112:0/253233713 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:02.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb327fff700 1 -- 172.21.15.112:0/253233713 >> v1:172.21.15.112:6789/0 conn(0x7fb3281387a0 legacy=0x7fb32813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb3277fe700 1 -- 172.21.15.112:0/253233713 >> v1:172.21.15.129:6789/0 conn(0x7fb328067680 legacy=0x7fb3280a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344832638 0 0) 0x7fb328067a50 con 0x7fb3281387a0 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient(hunting): my global_id is 134259 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb314003570 con 0x7fb3281387a0 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262080154 0 0) 0x7fb3280a1390 con 0x7fb328067680 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient(hunting): my global_id is 144148 2024-09-18T12:00:02.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb328067a50 con 0x7fb328067680 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3749269867 0 0) 0x7fb314003570 con 0x7fb3281387a0 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 >> v1:172.21.15.112:6790/0 conn(0x7fb328097a20 legacy=0x7fb3280a0c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 >> v1:172.21.15.129:6789/0 conn(0x7fb328067680 legacy=0x7fb3280a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 monclient: found mon.noname-a 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb328097fb0 con 0x7fb3281387a0 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: _finish_auth 0 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:02.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 5 monclient: authenticate success, global_id 134259 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb320001d80 con 0x7fb3281387a0 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: dump: 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:02.066 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb320001f60 con 0x7fb3281387a0 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 4 set_mon_vals no callback set 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 1 -- 172.21.15.112:0/253233713 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb320004e10 con 0x7fb3281387a0 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb326ffd700 10 monclient: dump: 2024-09-18T12:00:02.067 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 10 monclient: shutdown 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:02.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 -- 172.21.15.112:0/253233713 >> v1:172.21.15.112:6789/0 conn(0x7fb3281387a0 legacy=0x7fb32813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 -- 172.21.15.112:0/253233713 shutdown_connections 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 -- 172.21.15.112:0/253233713 >> 172.21.15.112:0/253233713 conn(0x7fb3280a5b80 msgr2=0x7fb3280a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 -- 172.21.15.112:0/253233713 shutdown_connections 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 -- 172.21.15.112:0/253233713 wait complete. 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 10 monclient: build_initial_monmap 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 10 monclient: monmap: 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:02.069 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 librados: starting msgr at 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 librados: starting objecter 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 Processor -- start 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 -- start start 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 librados: setting wanted keys 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 1 librados: calling monclient init 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 10 monclient: init 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.059+0000 7fb32de07700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient(hunting): picked mon.noname-b con 0x7fb328067680 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:02.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient(hunting): picked mon.noname-c con 0x7fb328097a20 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient(hunting): picked mon.noname-a con 0x7fb3281387a0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31ffff700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fb328097a20 legacy=0x7fb3281467a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb328137fb0 con 0x7fb3281387a0 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb328099920 con 0x7fb328067680 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32809a710 con 0x7fb328097a20 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient(hunting): authenticate will time out at 1773.904211s 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb327fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb3281387a0 0x7fb328149ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39824/0 (socket says 172.21.15.112:39824) 2024-09-18T12:00:02.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb327fff700 1 -- 172.21.15.112:0/788048657 learned_addr learned my addr 172.21.15.112:0/788048657 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:02.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb327fff700 1 -- 172.21.15.112:0/788048657 >> v1:172.21.15.112:6789/0 conn(0x7fb3281387a0 legacy=0x7fb328149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:02.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 client.?.objecter ms_handle_connect 0x7fb3281387a0 2024-09-18T12:00:02.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 client.?.objecter ms_handle_connect 0x7fb328067680 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb3277fe700 1 -- 172.21.15.112:0/788048657 >> v1:172.21.15.129:6789/0 conn(0x7fb328067680 legacy=0x7fb32809e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449446303 0 0) 0x7fb328137fb0 con 0x7fb3281387a0 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): my global_id is 134262 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb308004f10 con 0x7fb3281387a0 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762696577 0 0) 0x7fb328099920 con 0x7fb328067680 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:02.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): my global_id is 144151 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb328137fb0 con 0x7fb328067680 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2442158769 0 0) 0x7fb308004f10 con 0x7fb3281387a0 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb328099920 con 0x7fb3281387a0 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb320001b20 con 0x7fb3281387a0 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): dump: 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:02.076 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1112822152 0 0) 0x7fb328099920 con 0x7fb3281387a0 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 >> v1:172.21.15.112:6790/0 conn(0x7fb328097a20 legacy=0x7fb3281467a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 >> v1:172.21.15.129:6789/0 conn(0x7fb328067680 legacy=0x7fb32809e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 monclient: found mon.a 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:02.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32809acd0 con 0x7fb3281387a0 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient: _finish_auth 0 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 5 monclient: authenticate success, global_id 134262 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient: _renew_subs 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3280992f0 con 0x7fb3281387a0 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb320005340 con 0x7fb3281387a0 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 client.134262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb320005650 con 0x7fb3281387a0 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient: _renew_subs 2024-09-18T12:00:02.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb328137650 con 0x7fb3281387a0 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb325ffb700 4 set_mon_vals no callback set 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb32de07700 1 librados: init done 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.063+0000 7fb31f7fe700 10 monclient: dump: 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:02.079 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb32de07700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb32de07700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb32de07700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb328133cb0 con 0x7fb3281387a0 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb32000fed0 con 0x7fb3281387a0 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(627..627 src has 1..627) v4 ==== 5806+0+0 (unknown 3594278032 0 0) 0x7fb320044560 con 0x7fb3281387a0 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb31f7fe700 10 client.134262.objecter ms_dispatch 0x7fb328000fd0 osd_map(627..627 src has 1..627) v4 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb31f7fe700 3 client.134262.objecter handle_osd_map got epochs [627,627] > 0 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb31f7fe700 3 client.134262.objecter handle_osd_map decoding full epoch 627 2024-09-18T12:00:02.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb31f7fe700 20 client.134262.objecter dump_active .. 0 homeless 2024-09-18T12:00:02.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.067+0000 7fb3277fe700 10 monclient: get_auth_request con 0x7fb3080381c0 auth_method 0 2024-09-18T12:00:02.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.071+0000 7fb31f7fe700 10 client.134262.objecter ms_handle_connect 0x7fb3080381c0 2024-09-18T12:00:02.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.075+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fb320015050 con 0x7fb3281387a0 2024-09-18T12:00:02.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.075+0000 7fb31f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:02.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.075+0000 7fb31f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:02.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.295+0000 7fb32de07700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:00:02.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.295+0000 7fb32de07700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:00:02.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.295+0000 7fb32de07700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:02.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.295+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb32805f2e0 con 0x7fb3281387a0 2024-09-18T12:00:02.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.295+0000 7fb31f7fe700 1 -- 172.21.15.112:0/788048657 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 529128555) 0x7fb320014a50 con 0x7fb3281387a0 2024-09-18T12:00:02.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.295+0000 7fb31f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:00:02.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.295+0000 7fb31f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:02.298 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:00:02.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 10 librados: watch_flush enter 2024-09-18T12:00:02.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 10 librados: watch_flush exit 2024-09-18T12:00:02.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 20 client.134262.objecter shutdown clearing up homeless session... 2024-09-18T12:00:02.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 10 client.134262.objecter successfully canceled tick 2024-09-18T12:00:02.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 >> v1:172.21.15.129:6812/20806 conn(0x7fb3080381c0 legacy=0x7fb30803a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:02.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 10 monclient: shutdown 2024-09-18T12:00:02.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:02.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 >> v1:172.21.15.112:6789/0 conn(0x7fb3281387a0 legacy=0x7fb328149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:02.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 shutdown_connections 2024-09-18T12:00:02.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 >> 172.21.15.112:0/788048657 conn(0x7fb3280a5b80 msgr2=0x7fb328138b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:02.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 shutdown_connections 2024-09-18T12:00:02.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 1 -- 172.21.15.112:0/788048657 wait complete. 2024-09-18T12:00:02.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:02.299+0000 7fb32de07700 1 librados: shutdown 2024-09-18T12:00:05.313 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:00:05.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient: get_monmap_and_config 2024-09-18T12:00:05.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient: build_initial_monmap 2024-09-18T12:00:05.428 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient: monmap: 2024-09-18T12:00:05.428 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:05.429 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:05.429 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:05.429405+0000 2024-09-18T12:00:05.429 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:05.429405+0000 2024-09-18T12:00:05.429 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:05.429 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:05.429 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:05.429 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:05.429 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:05.430 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:05.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 1 Processor -- start 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 1 -- start start 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient: init 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient(hunting): picked mon.noname-b con 0x7fbe3006aa40 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient(hunting): picked mon.noname-c con 0x7fbe300a1e80 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient(hunting): picked mon.noname-a con 0x7fbe301386e0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe30097270 con 0x7fbe301386e0 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3006ae10 con 0x7fbe300a1e80 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe30097550 con 0x7fbe3006aa40 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:05.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe36be0700 10 monclient(hunting): authenticate will time out at 1777.270699s 2024-09-18T12:00:05.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe3497c700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fbe3006aa40 legacy=0x7fbe300a0a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:05.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe3517d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbe301386e0 0x7fbe3013aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39828/0 (socket says 172.21.15.112:39828) 2024-09-18T12:00:05.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.427+0000 7fbe3517d700 1 -- 172.21.15.112:0/1051330661 learned_addr learned my addr 172.21.15.112:0/1051330661 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:05.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe3517d700 1 -- 172.21.15.112:0/1051330661 >> v1:172.21.15.112:6789/0 conn(0x7fbe301386e0 legacy=0x7fbe3013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:05.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159759267 0 0) 0x7fbe30097270 con 0x7fbe301386e0 2024-09-18T12:00:05.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:05.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:05.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient(hunting): my global_id is 134265 2024-09-18T12:00:05.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe20003570 con 0x7fbe301386e0 2024-09-18T12:00:05.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2ffff700 1 -- 172.21.15.112:0/1051330661 >> v1:172.21.15.129:6789/0 conn(0x7fbe300a1e80 legacy=0x7fbe300a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:05.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773528061 0 0) 0x7fbe20003570 con 0x7fbe301386e0 2024-09-18T12:00:05.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 >> v1:172.21.15.112:6790/0 conn(0x7fbe3006aa40 legacy=0x7fbe300a0a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:00:05.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 >> v1:172.21.15.129:6789/0 conn(0x7fbe300a1e80 legacy=0x7fbe300a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:05.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:00:05.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 monclient: found mon.noname-a 2024-09-18T12:00:05.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:05.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:00:05.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30096a60 con 0x7fbe301386e0 2024-09-18T12:00:05.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: _finish_auth 0 2024-09-18T12:00:05.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:05.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbe28001d80 con 0x7fbe301386e0 2024-09-18T12:00:05.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 5 monclient: authenticate success, global_id 134265 2024-09-18T12:00:05.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: dump: 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:05.439 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe28001f60 con 0x7fbe301386e0 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 1 -- 172.21.15.112:0/1051330661 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbe28004e10 con 0x7fbe301386e0 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 4 set_mon_vals no callback set 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe2f7fe700 10 monclient: dump: 2024-09-18T12:00:05.440 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:05.441 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 10 monclient: shutdown 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 -- 172.21.15.112:0/1051330661 >> v1:172.21.15.112:6789/0 conn(0x7fbe301386e0 legacy=0x7fbe3013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 -- 172.21.15.112:0/1051330661 shutdown_connections 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 -- 172.21.15.112:0/1051330661 >> 172.21.15.112:0/1051330661 conn(0x7fbe300a5d10 msgr2=0x7fbe300a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 -- 172.21.15.112:0/1051330661 shutdown_connections 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 -- 172.21.15.112:0/1051330661 wait complete. 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 10 monclient: build_initial_monmap 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 10 monclient: monmap: 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:05.442 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:05.443 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:05.443 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:05.443 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:05.443 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:05.443 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:05.443 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:05.444 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 librados: starting msgr at 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 librados: starting objecter 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 Processor -- start 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 -- start start 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 librados: setting wanted keys 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 1 librados: calling monclient init 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.431+0000 7fbe36be0700 10 monclient: init 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 10 monclient(hunting): picked mon.noname-a con 0x7fbe3006aa40 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 10 monclient(hunting): picked mon.noname-c con 0x7fbe300a1e80 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 10 monclient(hunting): picked mon.noname-b con 0x7fbe301386e0 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe30097f80 con 0x7fbe3006aa40 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe30142640 con 0x7fbe301386e0 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe30141e70 con 0x7fbe300a1e80 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe36be0700 10 monclient(hunting): authenticate will time out at 1777.276610s 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2ffff700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fbe300a1e80 legacy=0x7fbe3009e2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe3497c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbe3006aa40 0x7fbe30141760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39838/0 (socket says 172.21.15.112:39838) 2024-09-18T12:00:05.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe3497c700 1 -- 172.21.15.112:0/177848016 learned_addr learned my addr 172.21.15.112:0/177848016 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:05.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe3497c700 1 -- 172.21.15.112:0/177848016 >> v1:172.21.15.112:6789/0 conn(0x7fbe3006aa40 legacy=0x7fbe30141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:05.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 client.?.objecter ms_handle_connect 0x7fbe3006aa40 2024-09-18T12:00:05.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:05.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190848695 0 0) 0x7fbe30097f80 con 0x7fbe3006aa40 2024-09-18T12:00:05.449 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 monclient(hunting): my global_id is 134268 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe14004f10 con 0x7fbe3006aa40 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 client.?.objecter ms_handle_connect 0x7fbe301386e0 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe3517d700 1 -- 172.21.15.112:0/177848016 >> v1:172.21.15.129:6789/0 conn(0x7fbe301386e0 legacy=0x7fbe30149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2389234968 0 0) 0x7fbe14004f10 con 0x7fbe3006aa40 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe30097f80 con 0x7fbe3006aa40 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbe1c0044e0 con 0x7fbe3006aa40 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 monclient(hunting): dump: 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:05.450 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.435+0000 7fbe2cff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1359981113 0 0) 0x7fbe30097f80 con 0x7fbe3006aa40 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 >> v1:172.21.15.112:6790/0 conn(0x7fbe300a1e80 legacy=0x7fbe3009e2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:00:05.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 >> v1:172.21.15.129:6789/0 conn(0x7fbe301386e0 legacy=0x7fbe30149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 10 monclient: _finish_hunting 0 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 monclient: found mon.a 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300a22f0 con 0x7fbe3006aa40 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 10 monclient: _finish_auth 0 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 5 monclient: authenticate success, global_id 134268 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 10 monclient: _renew_subs 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe300994c0 con 0x7fbe3006aa40 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 10 client.134268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 10 monclient: _renew_subs 2024-09-18T12:00:05.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe301373c0 con 0x7fbe3006aa40 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 1 librados: init done 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3005f2e0 con 0x7fbe3006aa40 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe1c0047b0 con 0x7fbe3006aa40 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbe1c004b40 con 0x7fbe3006aa40 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 10 monclient: dump: 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:05.453 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2effd700 4 set_mon_vals no callback set 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fbe1c004d20 con 0x7fbe3006aa40 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(627..627 src has 1..627) v4 ==== 5806+0+0 (unknown 3594278032 0 0) 0x7fbe1c0440d0 con 0x7fbe3006aa40 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 10 client.134268.objecter ms_dispatch 0x7fbe30000fd0 osd_map(627..627 src has 1..627) v4 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 3 client.134268.objecter handle_osd_map got epochs [627,627] > 0 2024-09-18T12:00:05.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 3 client.134268.objecter handle_osd_map decoding full epoch 627 2024-09-18T12:00:05.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2ffff700 10 monclient: get_auth_request con 0x7fbe140381e0 auth_method 0 2024-09-18T12:00:05.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.439+0000 7fbe2cff9700 20 client.134268.objecter dump_active .. 0 homeless 2024-09-18T12:00:05.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.443+0000 7fbe2cff9700 10 client.134268.objecter ms_handle_connect 0x7fbe140381e0 2024-09-18T12:00:05.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.447+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fbe1c005420 con 0x7fbe3006aa40 2024-09-18T12:00:05.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.447+0000 7fbe2cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:05.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.447+0000 7fbe2cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:05.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.671+0000 7fbe36be0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:00:05.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.671+0000 7fbe36be0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:00:05.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.671+0000 7fbe36be0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:05.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.671+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbe30063a80 con 0x7fbe3006aa40 2024-09-18T12:00:05.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.671+0000 7fbe2cff9700 1 -- 172.21.15.112:0/177848016 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1014382946) 0x7fbe1c014500 con 0x7fbe3006aa40 2024-09-18T12:00:05.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.671+0000 7fbe2cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:00:05.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.671+0000 7fbe2cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:05.674 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 10 librados: watch_flush enter 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 10 librados: watch_flush exit 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 20 client.134268.objecter shutdown clearing up homeless session... 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 10 client.134268.objecter successfully canceled tick 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 >> v1:172.21.15.129:6812/20806 conn(0x7fbe140381e0 legacy=0x7fbe1403a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 10 monclient: shutdown 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 >> v1:172.21.15.112:6789/0 conn(0x7fbe3006aa40 legacy=0x7fbe30141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 shutdown_connections 2024-09-18T12:00:05.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 >> 172.21.15.112:0/177848016 conn(0x7fbe300a5d10 msgr2=0x7fbe30137940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:05.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 shutdown_connections 2024-09-18T12:00:05.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 1 -- 172.21.15.112:0/177848016 wait complete. 2024-09-18T12:00:05.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:05.675+0000 7fbe36be0700 1 librados: shutdown 2024-09-18T12:00:08.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient: get_monmap_and_config 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient: build_initial_monmap 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient: monmap: 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:08.806271+0000 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:08.806271+0000 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:08.805 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:08.806 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:08.806 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:08.806 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:08.806 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 1 Processor -- start 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 1 -- start start 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient: init 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient(hunting): picked mon.noname-a con 0x7fcbb00a0380 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient(hunting): picked mon.noname-c con 0x7fcbb00a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient(hunting): picked mon.noname-b con 0x7fcbb0097970 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb0097550 con 0x7fcbb00a0380 2024-09-18T12:00:08.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb0096cb0 con 0x7fcbb00a0e60 2024-09-18T12:00:08.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb00a1440 con 0x7fcbb0097970 2024-09-18T12:00:08.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:08.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb5ad0700 10 monclient(hunting): authenticate will time out at 1780.647693s 2024-09-18T12:00:08.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb52cf700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcbb013aac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:08.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb4ace700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcbb00a0380 0x7fcbb00a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:39840/0 (socket says 172.21.15.112:39840) 2024-09-18T12:00:08.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.803+0000 7fcbb4ace700 1 -- 172.21.15.112:0/3637656729 learned_addr learned my addr 172.21.15.112:0/3637656729 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:08.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.807+0000 7fcbb4ace700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6789/0 conn(0x7fcbb00a0380 legacy=0x7fcbb00a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:08.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:08.807+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.129:6789/0 conn(0x7fcbb00a0e60 legacy=0x7fcbb0137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:09.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:09.007+0000 7fcbb52cf700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcbb013aac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:09.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:09.411+0000 7fcbb52cf700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcbb013aac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:10.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:10.215+0000 7fcbb52cf700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcbb013aac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:11.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 10 monclient(hunting): tick 2024-09-18T12:00:11.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 1 monclient(hunting): continuing hunt 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcbb013aac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.129:6789/0 conn(0x7fcbb00a0e60 legacy=0x7fcbb0137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6789/0 conn(0x7fcbb00a0380 legacy=0x7fcbb00a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 10 monclient(hunting): picked mon.noname-a con 0x7fcb9c003810 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fcbb0097970 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fcbb00a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb00a1de0 con 0x7fcb9c003810 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb00a1440 con 0x7fcbb00a0e60 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:11.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c010500 con 0x7fcbb0097970 2024-09-18T12:00:11.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbae7fc700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbb4ace700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6789/0 conn(0x7fcb9c003810 legacy=0x7fcb9c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:11.807+0000 7fcbb52cf700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.129:6789/0 conn(0x7fcbb00a0e60 legacy=0x7fcb9c00cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:12.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:12.011+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:12.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:12.415+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:13.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:13.218+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 10 monclient(hunting): tick 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 1 monclient(hunting): continuing hunt 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.129:6789/0 conn(0x7fcbb00a0e60 legacy=0x7fcb9c00cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6789/0 conn(0x7fcb9c003810 legacy=0x7fcb9c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fcb9c0107e0 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:14.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fcbb0097970 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:14.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 10 monclient(hunting): picked mon.noname-a con 0x7fcb9c003810 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:14.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c011760 con 0x7fcb9c003810 2024-09-18T12:00:14.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c010500 con 0x7fcb9c0107e0 2024-09-18T12:00:14.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 1 -- 172.21.15.112:0/3637656729 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c011510 con 0x7fcbb0097970 2024-09-18T12:00:14.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbae7fc700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:14.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:14.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbb52cf700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6789/0 conn(0x7fcb9c003810 legacy=0x7fcb9c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:14.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:14.806+0000 7fcbb4ace700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.129:6789/0 conn(0x7fcb9c0107e0 legacy=0x7fcb9c010bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:15.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:15.010+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:15.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:15.414+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:16.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.218+0000 7fcbaffff700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:16.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955542060 0 0) 0x7fcb9c010500 con 0x7fcb9c0107e0 2024-09-18T12:00:16.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:16.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:16.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient(hunting): my global_id is 144154 2024-09-18T12:00:16.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb98003570 con 0x7fcb9c0107e0 2024-09-18T12:00:16.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179760366 0 0) 0x7fcb98003570 con 0x7fcb9c0107e0 2024-09-18T12:00:16.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcb9c009500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:00:16.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.112:6789/0 conn(0x7fcb9c003810 legacy=0x7fcb9c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:00:16.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 monclient: found mon.noname-c 2024-09-18T12:00:16.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c00cdd0 con 0x7fcb9c0107e0 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: _finish_auth 0 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcba0001850 con 0x7fcb9c0107e0 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: dump: 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 5 monclient: authenticate success, global_id 144154 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcba00019b0 con 0x7fcb9c0107e0 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:16.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 1 -- 172.21.15.112:0/3637656729 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcba0001d40 con 0x7fcb9c0107e0 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 4 set_mon_vals no callback set 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbaf7fe700 10 monclient: dump: 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.586 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.586 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.586 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.586 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.586 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.587 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 10 monclient: shutdown 2024-09-18T12:00:16.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:16.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/3637656729 >> v1:172.21.15.129:6789/0 conn(0x7fcb9c0107e0 legacy=0x7fcb9c010bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/3637656729 shutdown_connections 2024-09-18T12:00:16.589 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.578+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/3637656729 >> 172.21.15.112:0/3637656729 conn(0x7fcbb0068100 msgr2=0x7fcbb00a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/3637656729 shutdown_connections 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/3637656729 wait complete. 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient: build_initial_monmap 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient: monmap: 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:16.590 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 librados: starting msgr at 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 librados: starting objecter 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 Processor -- start 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 -- start start 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 librados: setting wanted keys 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 librados: calling monclient init 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient: init 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient(hunting): picked mon.noname-a con 0x7fcb9c003810 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient(hunting): picked mon.noname-b con 0x7fcb9c0107e0 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient(hunting): picked mon.noname-c con 0x7fcbb0097970 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb011e9c0 con 0x7fcb9c003810 2024-09-18T12:00:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb011e370 con 0x7fcb9c0107e0 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb011e650 con 0x7fcbb0097970 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb5ad0700 10 monclient(hunting): authenticate will time out at 1788.427907s 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.582+0000 7fcbb52cf700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcbb0145e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.586+0000 7fcbb4ace700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcb9c003810 0x7fcbb013daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34858/0 (socket says 172.21.15.112:34858) 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.586+0000 7fcbb4ace700 1 -- 172.21.15.112:0/1039725050 learned_addr learned my addr 172.21.15.112:0/1039725050 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.586+0000 7fcbb4ace700 1 -- 172.21.15.112:0/1039725050 >> v1:172.21.15.112:6789/0 conn(0x7fcb9c003810 legacy=0x7fcbb013daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.586+0000 7fcbacff9700 10 client.?.objecter ms_handle_connect 0x7fcb9c003810 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.586+0000 7fcbacff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.586+0000 7fcbacff9700 10 client.?.objecter ms_handle_connect 0x7fcb9c0107e0 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.586+0000 7fcbacff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.586+0000 7fcbaffff700 1 -- 172.21.15.112:0/1039725050 >> v1:172.21.15.129:6789/0 conn(0x7fcb9c0107e0 legacy=0x7fcbb0142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:16.594 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267277652 0 0) 0x7fcbb011e9c0 con 0x7fcb9c003810 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient(hunting): my global_id is 134271 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c010500 con 0x7fcb9c003810 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3400511584 0 0) 0x7fcb9c010500 con 0x7fcb9c003810 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbb011e9c0 con 0x7fcb9c003810 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcba0001850 con 0x7fcb9c003810 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient(hunting): dump: 2024-09-18T12:00:16.595 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.596 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.596 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.596 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.596 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.596 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.597 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.597 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.597 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.597 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:16.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4082190148 0 0) 0x7fcbb011e9c0 con 0x7fcb9c003810 2024-09-18T12:00:16.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 >> v1:172.21.15.112:6790/0 conn(0x7fcbb0097970 legacy=0x7fcbb0145e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:00:16.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 >> v1:172.21.15.129:6789/0 conn(0x7fcb9c0107e0 legacy=0x7fcbb0142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient: _finish_hunting 0 2024-09-18T12:00:16.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 monclient: found mon.a 2024-09-18T12:00:16.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:16.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb013e200 con 0x7fcb9c003810 2024-09-18T12:00:16.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient: _finish_auth 0 2024-09-18T12:00:16.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:16.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 5 monclient: authenticate success, global_id 134271 2024-09-18T12:00:16.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 10 monclient: _renew_subs 2024-09-18T12:00:16.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb0134810 con 0x7fcb9c003810 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 10 client.134271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 10 monclient: _renew_subs 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb011d6a0 con 0x7fcb9c003810 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbb5ad0700 1 librados: init done 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcba00013e0 con 0x7fcb9c003810 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcba0004e80 con 0x7fcb9c003810 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbacff9700 10 monclient: dump: 2024-09-18T12:00:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.590+0000 7fcbaeffd700 4 set_mon_vals no callback set 2024-09-18T12:00:16.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbb5ad0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:16.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbb5ad0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbb5ad0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb900050b0 con 0x7fcb9c003810 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fcba0005060 con 0x7fcb9c003810 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(628..628 src has 1..628) v4 ==== 5801+0+0 (unknown 4191963335 0 0) 0x7fcba0043fe0 con 0x7fcb9c003810 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbacff9700 10 client.134271.objecter ms_dispatch 0x7fcbb0000fd0 osd_map(628..628 src has 1..628) v4 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbacff9700 3 client.134271.objecter handle_osd_map got epochs [628,628] > 0 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbacff9700 3 client.134271.objecter handle_osd_map decoding full epoch 628 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbacff9700 20 client.134271.objecter dump_active .. 0 homeless 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbaffff700 10 monclient: get_auth_request con 0x7fcb9c0390f0 auth_method 0 2024-09-18T12:00:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.594+0000 7fcbacff9700 10 client.134271.objecter ms_handle_connect 0x7fcb9c0390f0 2024-09-18T12:00:16.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.602+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fcba0014ad0 con 0x7fcb9c003810 2024-09-18T12:00:16.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.602+0000 7fcbacff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:16.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.602+0000 7fcbacff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:16.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:00:16.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:00:16.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcb900058e0 con 0x7fcb9c003810 2024-09-18T12:00:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbacff9700 1 -- 172.21.15.112:0/1039725050 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3568273063) 0x7fcba00144d0 con 0x7fcb9c003810 2024-09-18T12:00:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbacff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:00:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbacff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:16.832 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":74,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 10 librados: watch_flush enter 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 10 librados: watch_flush exit 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 20 client.134271.objecter shutdown clearing up homeless session... 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 10 client.134271.objecter successfully canceled tick 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 >> v1:172.21.15.129:6812/20806 conn(0x7fcb9c0390f0 legacy=0x7fcb9c03b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 10 monclient: shutdown 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 >> v1:172.21.15.112:6789/0 conn(0x7fcb9c003810 legacy=0x7fcbb013daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 shutdown_connections 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.830+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 >> 172.21.15.112:0/1039725050 conn(0x7fcbb0068100 msgr2=0x7fcbb00a4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.834+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 shutdown_connections 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.834+0000 7fcbb5ad0700 1 -- 172.21.15.112:0/1039725050 wait complete. 2024-09-18T12:00:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.834+0000 7fcbb5ad0700 1 librados: shutdown 2024-09-18T12:00:16.847 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:00:16.847 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:00:16.847 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:00:16.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient: get_monmap_and_config 2024-09-18T12:00:16.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient: build_initial_monmap 2024-09-18T12:00:16.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient: monmap: 2024-09-18T12:00:16.957 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:16.957 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.957 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:16.957960+0000 2024-09-18T12:00:16.957 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:16.957960+0000 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 1 Processor -- start 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 1 -- start start 2024-09-18T12:00:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient: init 2024-09-18T12:00:16.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:16.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:00:16.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient(hunting): picked mon.noname-a con 0x7f893c0a2040 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient(hunting): picked mon.noname-c con 0x7f893c12edc0 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient(hunting): picked mon.noname-b con 0x7f893c12f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c0981a0 con 0x7f893c0a2040 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c06c120 con 0x7f893c12edc0 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c06eef0 con 0x7f893c12f8a0 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f8940c8e700 10 monclient(hunting): authenticate will time out at 1788.799080s 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f893bfff700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f893c12f8a0 legacy=0x7f893c137230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f893b7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f893c0a2040 0x7f893c097a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34874/0 (socket says 172.21.15.112:34874) 2024-09-18T12:00:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.954+0000 7f893b7fe700 1 -- 172.21.15.112:0/485852170 learned_addr learned my addr 172.21.15.112:0/485852170 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:16.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893b7fe700 1 -- 172.21.15.112:0/485852170 >> v1:172.21.15.112:6789/0 conn(0x7f893c0a2040 legacy=0x7f893c097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:16.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893affd700 1 -- 172.21.15.112:0/485852170 >> v1:172.21.15.129:6789/0 conn(0x7f893c12edc0 legacy=0x7f893c12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045471169 0 0) 0x7f893c0981a0 con 0x7f893c0a2040 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient(hunting): my global_id is 134274 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8924003570 con 0x7f893c0a2040 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981798501 0 0) 0x7f893c06c120 con 0x7f893c12edc0 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient(hunting): my global_id is 144157 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f893c0981a0 con 0x7f893c12edc0 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2303253190 0 0) 0x7f8924003570 con 0x7f893c0a2040 2024-09-18T12:00:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 >> v1:172.21.15.112:6790/0 conn(0x7f893c12f8a0 legacy=0x7f893c137230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 >> v1:172.21.15.129:6789/0 conn(0x7f893c12edc0 legacy=0x7f893c12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 monclient: found mon.noname-a 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c096dd0 con 0x7f893c0a2040 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: _finish_auth 0 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 5 monclient: authenticate success, global_id 134274 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8930001a00 con 0x7f893c0a2040 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: dump: 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8930001be0 con 0x7f893c0a2040 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 4 set_mon_vals no callback set 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 1 -- 172.21.15.112:0/485852170 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8930004a90 con 0x7f893c0a2040 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f893a7fc700 10 monclient: dump: 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 10 monclient: shutdown 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 1 -- 172.21.15.112:0/485852170 >> v1:172.21.15.112:6789/0 conn(0x7f893c0a2040 legacy=0x7f893c097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 1 -- 172.21.15.112:0/485852170 shutdown_connections 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 1 -- 172.21.15.112:0/485852170 >> 172.21.15.112:0/485852170 conn(0x7f893c067da0 msgr2=0x7f893c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 1 -- 172.21.15.112:0/485852170 shutdown_connections 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.958+0000 7f8940c8e700 1 -- 172.21.15.112:0/485852170 wait complete. 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient: build_initial_monmap 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient: monmap: 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 librados: starting msgr at 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 librados: starting objecter 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 Processor -- start 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 -- start start 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 librados: setting wanted keys 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 librados: calling monclient init 2024-09-18T12:00:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient: init 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient(hunting): picked mon.noname-b con 0x7f893c0a2040 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient(hunting): picked mon.noname-c con 0x7f893c12edc0 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient(hunting): picked mon.noname-a con 0x7f893c12f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c12ad30 con 0x7f893c12f8a0 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c142800 con 0x7f893c0a2040 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f893affd700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f893c12edc0 legacy=0x7f893c1420f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c129d60 con 0x7f893c12edc0 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f893bfff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f893c12f8a0 0x7f893c149c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34876/0 (socket says 172.21.15.112:34876) 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f893bfff700 1 -- 172.21.15.112:0/1399210490 learned_addr learned my addr 172.21.15.112:0/1399210490 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.962+0000 7f8940c8e700 10 monclient(hunting): authenticate will time out at 1788.806600s 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f893bfff700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6789/0 conn(0x7f893c12f8a0 legacy=0x7f893c149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 client.?.objecter ms_handle_connect 0x7f893c12f8a0 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288239303 0 0) 0x7f893c12ad30 con 0x7f893c12f8a0 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): my global_id is 134277 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8928004f10 con 0x7f893c12f8a0 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 client.?.objecter ms_handle_connect 0x7f893c0a2040 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f893b7fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.129:6789/0 conn(0x7f893c0a2040 legacy=0x7f893c1419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404593284 0 0) 0x7f893c142800 con 0x7f893c0a2040 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): my global_id is 144160 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f893c12ad30 con 0x7f893c0a2040 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3917718114 0 0) 0x7f8928004f10 con 0x7f893c12f8a0 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f893c142800 con 0x7f893c12f8a0 2024-09-18T12:00:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8934004480 con 0x7f893c12f8a0 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): dump: 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399356072 0 0) 0x7f893c12ad30 con 0x7f893c0a2040 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f893b7fe700 1 --1- 172.21.15.112:0/1399210490 >> v1:172.21.15.129:6789/0 conn(0x7f893c0a2040 0x7f893c1419e0 :-1 s=OPENED pgs=9220 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f8930004830 mon_map magic: 0 v1 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8928004f10 con 0x7f893c0a2040 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 145462538 0 0) 0x7f893c142800 con 0x7f893c12f8a0 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6790/0 conn(0x7f893c12edc0 legacy=0x7f893c1420f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.129:6789/0 conn(0x7f893c0a2040 legacy=0x7f893c1419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient: _finish_hunting 0 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 monclient: found mon.a 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c12b280 con 0x7f893c12f8a0 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient: _finish_auth 0 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8930004830 con 0x7f893c0a2040 2024-09-18T12:00:16.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 5 monclient: authenticate success, global_id 134277 2024-09-18T12:00:16.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 10 monclient: _renew_subs 2024-09-18T12:00:16.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c09e380 con 0x7f893c12f8a0 2024-09-18T12:00:16.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T12:00:16.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8934004660 con 0x7f893c12f8a0 2024-09-18T12:00:16.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:16.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f89340049f0 con 0x7f893c12f8a0 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 10 client.134277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 monclient: dump: 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 10 monclient: _renew_subs 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c12bf50 con 0x7f893c12f8a0 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 1 librados: init done 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8939ffb700 4 set_mon_vals no callback set 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f893c12b0b0 con 0x7f893c09d690 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8934004bd0 con 0x7f893c12f8a0 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(628..628 src has 1..628) v4 ==== 5801+0+0 (unknown 4191963335 0 0) 0x7f8934043fd0 con 0x7f893c12f8a0 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 10 client.134277.objecter ms_dispatch 0x7f893c000fd0 osd_map(628..628 src has 1..628) v4 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 3 client.134277.objecter handle_osd_map got epochs [628,628] > 0 2024-09-18T12:00:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 3 client.134277.objecter handle_osd_map decoding full epoch 628 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f89237fe700 20 client.134277.objecter dump_active .. 0 homeless 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.966+0000 7f893affd700 10 monclient: get_auth_request con 0x7f8928038310 auth_method 0 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f893b7fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6789/0 conn(0x7f893c09d690 legacy=0x7f893c098360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 client.134277.objecter ms_handle_connect 0x7f893c09d690 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 client.134277.objecter resend_mon_ops 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959324580 0 0) 0x7f893c12b0b0 con 0x7f893c09d690 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 monclient(hunting): my global_id is 134280 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f893c142800 con 0x7f893c09d690 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1383740050 0 0) 0x7f893c142800 con 0x7f893c09d690 2024-09-18T12:00:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f893c12b0b0 con 0x7f893c09d690 2024-09-18T12:00:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8930004f60 con 0x7f893c09d690 2024-09-18T12:00:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 monclient: dump: 2024-09-18T12:00:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1385349008 0 0) 0x7f893c12b0b0 con 0x7f893c09d690 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f893c09da60 con 0x7f893c09d690 2024-09-18T12:00:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 client.134277.objecter ms_handle_connect 0x7f8928038310 2024-09-18T12:00:16.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f8930007100 con 0x7f893c09d690 2024-09-18T12:00:16.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:16.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:16.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6789/0 conn(0x7f893c09d690 legacy=0x7f893c098360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.970+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6789/0 conn(0x7f893c09d690 legacy=0x7f893c098360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:16.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f8940c8e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f893c05f2e0 con 0x7f893c02ad10 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f893b7fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6789/0 conn(0x7f893c02ad10 legacy=0x7f893c098a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 10 client.134277.objecter ms_handle_connect 0x7f893c02ad10 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 10 client.134277.objecter resend_mon_ops 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361929774 0 0) 0x7f893c05f2e0 con 0x7f893c02ad10 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 10 monclient(hunting): my global_id is 134283 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f893c12b0b0 con 0x7f893c02ad10 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1440210938 0 0) 0x7f893c12b0b0 con 0x7f893c02ad10 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f893c05f2e0 con 0x7f893c02ad10 2024-09-18T12:00:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8930005640 con 0x7f893c02ad10 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.982+0000 7f89237fe700 10 monclient: dump: 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3022980864 0 0) 0x7f893c05f2e0 con 0x7f893c02ad10 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f893c0a1360 con 0x7f893c02ad10 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4162 (unknown 0 0 4136419737) 0x7f89300092f0 con 0x7f893c02ad10 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f89237fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f89237fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6789/0 conn(0x7f893c02ad10 legacy=0x7f893c098a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f89237fe700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6789/0 conn(0x7f893c02ad10 legacy=0x7f893c098a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:00:16.991 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 74, 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: 1 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:16.992 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:16.993 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:00:16.994 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:00:16.995 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:00:16.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:00:16.997 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:16.998 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:16.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:17.001 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:17.002 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.003 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:00:17.003 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 10 librados: watch_flush enter 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 10 librados: watch_flush exit 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 20 client.134277.objecter shutdown clearing up homeless session... 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 10 client.134277.objecter successfully canceled tick 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.129:6812/20806 conn(0x7f8928038310 legacy=0x7f892803a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 10 monclient: shutdown 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f893bfff700 1 -- 172.21.15.112:0/1399210490 reap_dead start 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 >> v1:172.21.15.112:6789/0 conn(0x7f893c12f8a0 legacy=0x7f893c149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 shutdown_connections 2024-09-18T12:00:17.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 >> 172.21.15.112:0/1399210490 conn(0x7f893c067da0 msgr2=0x7f893c09f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:17.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 shutdown_connections 2024-09-18T12:00:17.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 1 -- 172.21.15.112:0/1399210490 wait complete. 2024-09-18T12:00:17.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:16.986+0000 7f8940c8e700 1 librados: shutdown 2024-09-18T12:00:17.006 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:00:17.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:00:17.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:00:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.114+0000 7f67fffff700 10 monclient: get_monmap_and_config 2024-09-18T12:00:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.114+0000 7f67fffff700 10 monclient: build_initial_monmap 2024-09-18T12:00:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.114+0000 7f67fffff700 10 monclient: monmap: 2024-09-18T12:00:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:00:17.120335+0000 2024-09-18T12:00:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:00:17.120335+0000 2024-09-18T12:00:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:00:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:00:17.120 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 1 Processor -- start 2024-09-18T12:00:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 1 -- start start 2024-09-18T12:00:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 10 monclient: init 2024-09-18T12:00:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:00:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 10 monclient(hunting): picked mon.noname-c con 0x7f68000a2040 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 10 monclient(hunting): picked mon.noname-b con 0x7f680012edc0 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 10 monclient(hunting): picked mon.noname-a con 0x7f680012f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68000981a0 con 0x7f680012f8a0 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680006c120 con 0x7f68000a2040 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680006eef0 con 0x7f680012edc0 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fffff700 10 monclient(hunting): authenticate will time out at 1788.961282s 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67fe7fc700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f680012edc0 legacy=0x7f680012f190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67ff7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f680012f8a0 0x7f6800137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34910/0 (socket says 172.21.15.112:34910) 2024-09-18T12:00:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67ff7fe700 1 -- 172.21.15.112:0/1691552049 learned_addr learned my addr 172.21.15.112:0/1691552049 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.118+0000 7f67ff7fe700 1 -- 172.21.15.112:0/1691552049 >> v1:172.21.15.112:6789/0 conn(0x7f680012f8a0 legacy=0x7f6800137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67feffd700 1 -- 172.21.15.112:0/1691552049 >> v1:172.21.15.129:6789/0 conn(0x7f68000a2040 legacy=0x7f6800097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755047169 0 0) 0x7f68000981a0 con 0x7f680012f8a0 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient(hunting): my global_id is 134286 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67ec003570 con 0x7f680012f8a0 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162205196 0 0) 0x7f680006c120 con 0x7f68000a2040 2024-09-18T12:00:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient(hunting): my global_id is 144163 2024-09-18T12:00:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68000981a0 con 0x7f68000a2040 2024-09-18T12:00:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 750327477 0 0) 0x7f67ec003570 con 0x7f680012f8a0 2024-09-18T12:00:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 >> v1:172.21.15.112:6790/0 conn(0x7f680012edc0 legacy=0x7f680012f190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:00:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 >> v1:172.21.15.129:6789/0 conn(0x7f68000a2040 legacy=0x7f6800097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: _finish_hunting 0 2024-09-18T12:00:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 monclient: found mon.noname-a 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800096dd0 con 0x7f680012f8a0 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: _finish_auth 0 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 5 monclient: authenticate success, global_id 134286 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67f0001a00 con 0x7f680012f8a0 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: dump: 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f0001be0 con 0x7f680012f8a0 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 10 monclient: get_monmap_and_config success 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 1 -- 172.21.15.112:0/1691552049 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67f0004a90 con 0x7f680012f8a0 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 4 set_mon_vals no callback set 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fdffb700 10 monclient: dump: 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:17.129 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 10 monclient: shutdown 2024-09-18T12:00:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 -- 172.21.15.112:0/1691552049 >> v1:172.21.15.112:6789/0 conn(0x7f680012f8a0 legacy=0x7f6800137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 -- 172.21.15.112:0/1691552049 shutdown_connections 2024-09-18T12:00:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 -- 172.21.15.112:0/1691552049 >> 172.21.15.112:0/1691552049 conn(0x7f6800067da0 msgr2=0x7f68000a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 -- 172.21.15.112:0/1691552049 shutdown_connections 2024-09-18T12:00:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 -- 172.21.15.112:0/1691552049 wait complete. 2024-09-18T12:00:17.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 10 monclient: build_initial_monmap 2024-09-18T12:00:17.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 10 monclient: monmap: 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 librados: starting msgr at 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 librados: starting objecter 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 Processor -- start 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 -- start start 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 librados: setting wanted keys 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 1 librados: calling monclient init 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 10 monclient: init 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 10 monclient: _reopen_session rank -1 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.122+0000 7f67fffff700 10 monclient(hunting): picked mon.noname-b con 0x7f68000a2040 addr v1:172.21.15.129:6789/0 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 monclient(hunting): picked mon.noname-c con 0x7f680012edc0 addr v1:172.21.15.112:6790/0 2024-09-18T12:00:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 monclient(hunting): picked mon.noname-a con 0x7f680012f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680006f110 con 0x7f680012f8a0 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800138290 con 0x7f68000a2040 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fe7fc700 1 -- >> v1:172.21.15.112:6790/0 conn(0x7f680012edc0 legacy=0x7f68001348f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6790/0 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800135000 con 0x7f680012edc0 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 monclient(hunting): _renew_subs 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 monclient(hunting): authenticate will time out at 1788.968124s 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67ff7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f680012f8a0 0x7f6800149e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34914/0 (socket says 172.21.15.112:34914) 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67ff7fe700 1 -- 172.21.15.112:0/1337281707 learned_addr learned my addr 172.21.15.112:0/1337281707 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:00:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67ff7fe700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.112:6789/0 conn(0x7f680012f8a0 legacy=0x7f6800149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 client.?.objecter ms_handle_connect 0x7f680012f8a0 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386778942 0 0) 0x7f680006f110 con 0x7f680012f8a0 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): my global_id is 134289 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e0004f10 con 0x7f680012f8a0 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 client.?.objecter ms_handle_connect 0x7f68000a2040 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67feffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6789/0 conn(0x7f68000a2040 legacy=0x7f68001341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3772173005 0 0) 0x7f67e0004f10 con 0x7f680012f8a0 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f680006f110 con 0x7f680012f8a0 2024-09-18T12:00:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67f0004830 con 0x7f680012f8a0 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): dump: 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690561344 0 0) 0x7f6800138290 con 0x7f68000a2040 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient(hunting): my global_id is 144166 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e0004f10 con 0x7f68000a2040 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2292861770 0 0) 0x7f680006f110 con 0x7f680012f8a0 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.112:6790/0 conn(0x7f680012edc0 legacy=0x7f68001348f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6789/0 conn(0x7f68000a2040 legacy=0x7f68001341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 monclient: found mon.a 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:00:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680006ce80 con 0x7f680012f8a0 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient: _finish_auth 0 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 5 monclient: authenticate success, global_id 134289 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 monclient: _renew_subs 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6800136680 con 0x7f680012f8a0 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 client.134289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 monclient: _renew_subs 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6800137420 con 0x7f680012f8a0 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fffff700 1 librados: init done 2024-09-18T12:00:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67ff7fe700 1 --1- 172.21.15.112:0/1337281707 >> v1:172.21.15.112:6789/0 conn(0x7f680012f8a0 0x7f6800149e80 :-1 s=OPENED pgs=665 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f67f0004af0 config(0 keys) v1 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f0004af0 con 0x7f680012f8a0 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67f6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.126+0000 7f67fd7fa700 4 set_mon_vals no callback set 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67f0005160 con 0x7f680012f8a0 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67f6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67f6ffd700 10 monclient: dump: 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f67f000f9e0 con 0x7f680012f8a0 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67fffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f67e4006590 con 0x7f67e4002840 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67fe7fc700 10 monclient: get_auth_request con 0x7f67e00382e0 auth_method 0 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67feffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6789/0 conn(0x7f67e4002840 legacy=0x7f67e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67f6ffd700 10 client.134289.objecter ms_handle_connect 0x7f67e4002840 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.130+0000 7f67f6ffd700 10 client.134289.objecter resend_mon_ops 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651836247 0 0) 0x7f67e4006590 con 0x7f67e4002840 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 monclient(hunting): my global_id is 144169 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f680006f110 con 0x7f67e4002840 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 client.134289.objecter ms_handle_connect 0x7f67e00382e0 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4151515213 0 0) 0x7f680006f110 con 0x7f67e4002840 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67e4006590 con 0x7f67e4002840 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67e8001b00 con 0x7f67e4002840 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 monclient: dump: 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:17.146 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:17.146 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:17.146 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:17.146 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4164452545 0 0) 0x7f67e4006590 con 0x7f67e4002840 2024-09-18T12:00:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67e400a140 con 0x7f67e4002840 2024-09-18T12:00:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f67e8005810 con 0x7f67e4002840 2024-09-18T12:00:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:00:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:00:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6789/0 conn(0x7f67e4002840 legacy=0x7f67e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.134+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6789/0 conn(0x7f67e4002840 legacy=0x7f67e4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:17.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.146+0000 7f67fffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:00:17.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.146+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f67e4009670 con 0x7f67e40090e0 2024-09-18T12:00:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 10 client.134289.objecter ms_handle_connect 0x7f67e40090e0 2024-09-18T12:00:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 10 client.134289.objecter resend_mon_ops 2024-09-18T12:00:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67feffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6789/0 conn(0x7f67e40090e0 legacy=0x7f67e4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:00:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671017298 0 0) 0x7f67e4009670 con 0x7f67e40090e0 2024-09-18T12:00:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:00:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:00:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 10 monclient(hunting): my global_id is 144172 2024-09-18T12:00:17.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e4006590 con 0x7f67e40090e0 2024-09-18T12:00:17.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2568045733 0 0) 0x7f67e4006590 con 0x7f67e40090e0 2024-09-18T12:00:17.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67e4009670 con 0x7f67e40090e0 2024-09-18T12:00:17.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f67e8002290 con 0x7f67e40090e0 2024-09-18T12:00:17.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 10 monclient: dump: 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1959523931 0 0) 0x7f67e4009670 con 0x7f67e40090e0 2024-09-18T12:00:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.150+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f67e4009930 con 0x7f67e40090e0 2024-09-18T12:00:17.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3990 (unknown 0 0 336667586) 0x7f67e8005aa0 con 0x7f67e40090e0 2024-09-18T12:00:17.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67f6ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:00:17.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67f6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:00:17.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6789/0 conn(0x7f67e40090e0 legacy=0x7f67e4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6789/0 conn(0x7f67e40090e0 legacy=0x7f67e4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:00:17.157 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:00:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:00:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:00:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:00:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 74, 2024-09-18T12:00:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:00:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:00:17.157 INFO:teuthology.orchestra.run.smithi112.stdout: 1 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:17.158 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:17.159 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:00:17.160 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:00:17.161 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.162 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.163 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:00:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.164 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:00:17.165 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:00:17.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:00:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.167 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:00:17.168 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:00:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:00:17.169 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:00:17.169 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:00:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 10 librados: watch_flush enter 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 10 librados: watch_flush exit 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 20 client.134289.objecter shutdown clearing up homeless session... 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 10 client.134289.objecter successfully canceled tick 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.129:6812/20806 conn(0x7f67e00382e0 legacy=0x7f67e003a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 10 monclient: shutdown 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 >> v1:172.21.15.112:6789/0 conn(0x7f680012f8a0 legacy=0x7f6800149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67ff7fe700 1 -- 172.21.15.112:0/1337281707 reap_dead start 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67ff7fe700 1 -- 172.21.15.112:0/1337281707 reap_dead start 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67f6ffd700 1 -- 172.21.15.112:0/1337281707 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(628..628 src has 1..628) v4 ==== 5801+0+0 (unknown 4191963335 0 0) 0x7f67f0044270 con 0x7f680012f8a0 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67f6ffd700 10 client.134289.objecter ms_dispatch 0x7f6800000fd0 osd_map(628..628 src has 1..628) v4 2024-09-18T12:00:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67f6ffd700 10 librados: disconnected, discarding osd_map(628..628 src has 1..628) v4 2024-09-18T12:00:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 shutdown_connections 2024-09-18T12:00:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 >> 172.21.15.112:0/1337281707 conn(0x7f6800067da0 msgr2=0x7f680009fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:00:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 shutdown_connections 2024-09-18T12:00:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 1 -- 172.21.15.112:0/1337281707 wait complete. 2024-09-18T12:00:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:00:17.154+0000 7f67fffff700 1 librados: shutdown 2024-09-18T12:00:17.172 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:00:17.172 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T12:01:47.173 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:01:47.174 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:01:47.174 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:01:47.174 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:01:47.178 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:01:47.178 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:01:47.178 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:01:47.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.276+0000 7f8e00e75700 10 monclient: get_monmap_and_config 2024-09-18T12:01:47.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.276+0000 7f8e00e75700 10 monclient: build_initial_monmap 2024-09-18T12:01:47.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.276+0000 7f8e00e75700 10 monclient: monmap: 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:47.283189+0000 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:47.283189+0000 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.276+0000 7f8e00e75700 1 Processor -- start 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.276+0000 7f8e00e75700 1 -- start start 2024-09-18T12:01:47.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.276+0000 7f8e00e75700 10 monclient: init 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.276+0000 7f8e00e75700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 10 monclient(hunting): picked mon.noname-b con 0x7f8dfc06a780 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 10 monclient(hunting): picked mon.noname-c con 0x7f8dfc06ab50 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 10 monclient(hunting): picked mon.noname-a con 0x7f8dfc135590 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc09fa00 con 0x7f8dfc135590 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc06e280 con 0x7f8dfc06ab50 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc09fc40 con 0x7f8dfc06a780 2024-09-18T12:01:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 10 monclient(hunting): authenticate will time out at 1879.124206s 2024-09-18T12:01:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8dfa59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8dfc06a780 0x7f8dfc06db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42578/0 (socket says 172.21.15.112:42578) 2024-09-18T12:01:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8dfa59c700 1 -- 172.21.15.112:0/3665621536 learned_addr learned my addr 172.21.15.112:0/3665621536 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8dfad9d700 1 --1- 172.21.15.112:0/3665621536 >> v1:172.21.15.112:6789/0 conn(0x7f8dfc135590 0x7f8dfc137970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56826/0 (socket says 172.21.15.112:56826) 2024-09-18T12:01:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8dfa59c700 1 -- 172.21.15.112:0/3665621536 >> v1:172.21.15.112:6790/0 conn(0x7f8dfc06a780 legacy=0x7f8dfc06db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8dfad9d700 1 -- 172.21.15.112:0/3665621536 >> v1:172.21.15.112:6789/0 conn(0x7f8dfc135590 legacy=0x7f8dfc137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606419289 0 0) 0x7f8dfc09fa00 con 0x7f8dfc135590 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient(hunting): my global_id is 134292 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de4003570 con 0x7f8dfc135590 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418484273 0 0) 0x7f8de4003570 con 0x7f8dfc135590 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 >> v1:172.21.15.112:6790/0 conn(0x7f8dfc06a780 legacy=0x7f8dfc06db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 >> v1:172.21.15.129:6789/0 conn(0x7f8dfc06ab50 legacy=0x7f8dfc06afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: _finish_hunting 0 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 monclient: found mon.noname-a 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:01:47.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc06c7a0 con 0x7f8dfc135590 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: _finish_auth 0 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8dec001940 con 0x7f8dfc135590 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 5 monclient: authenticate success, global_id 134292 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: dump: 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.289 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dec001c10 con 0x7f8dfc135590 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8e00e75700 4 set_mon_vals no callback set 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 1 -- 172.21.15.112:0/3665621536 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8dec0049d0 con 0x7f8dfc135590 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:47.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.280+0000 7f8df959a700 10 monclient: dump: 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 10 monclient: shutdown 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 -- 172.21.15.112:0/3665621536 >> v1:172.21.15.112:6789/0 conn(0x7f8dfc135590 legacy=0x7f8dfc137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 -- 172.21.15.112:0/3665621536 shutdown_connections 2024-09-18T12:01:47.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 -- 172.21.15.112:0/3665621536 >> 172.21.15.112:0/3665621536 conn(0x7f8dfc0680d0 msgr2=0x7f8dfc0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:47.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 -- 172.21.15.112:0/3665621536 shutdown_connections 2024-09-18T12:01:47.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 -- 172.21.15.112:0/3665621536 wait complete. 2024-09-18T12:01:47.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 10 monclient: build_initial_monmap 2024-09-18T12:01:47.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 10 monclient: monmap: 2024-09-18T12:01:47.293 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:47.293 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:47.293 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 librados: starting msgr at 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 librados: starting objecter 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 Processor -- start 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 -- start start 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 librados: setting wanted keys 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 1 librados: calling monclient init 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.284+0000 7f8e00e75700 10 monclient: init 2024-09-18T12:01:47.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 10 monclient(hunting): picked mon.noname-b con 0x7f8dfc06a780 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 10 monclient(hunting): picked mon.noname-a con 0x7f8dfc06ab50 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 10 monclient(hunting): picked mon.noname-c con 0x7f8dfc135590 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc09bce0 con 0x7f8dfc06ab50 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc09b200 con 0x7f8dfc06a780 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc11f1e0 con 0x7f8dfc135590 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dfad9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f8dfc135590 0x7f8dfc11c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42584/0 (socket says 172.21.15.112:42584) 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dfad9d700 1 -- 172.21.15.112:0/3719251211 learned_addr learned my addr 172.21.15.112:0/3719251211 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8df9d9b700 1 --1- 172.21.15.112:0/3719251211 >> v1:172.21.15.112:6789/0 conn(0x7f8dfc06ab50 0x7f8dfc11c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56840/0 (socket says 172.21.15.112:56840) 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8e00e75700 10 monclient(hunting): authenticate will time out at 1879.133502s 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dfad9d700 1 -- 172.21.15.112:0/3719251211 >> v1:172.21.15.112:6790/0 conn(0x7f8dfc135590 legacy=0x7f8dfc11c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dea7fc700 10 client.?.objecter ms_handle_connect 0x7f8dfc135590 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dea7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dea7fc700 10 client.?.objecter ms_handle_connect 0x7f8dfc06ab50 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dea7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8df9d9b700 1 -- 172.21.15.112:0/3719251211 >> v1:172.21.15.112:6789/0 conn(0x7f8dfc06ab50 legacy=0x7f8dfc11c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dea7fc700 10 client.?.objecter ms_handle_connect 0x7f8dfc06a780 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dea7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.288+0000 7f8dfa59c700 1 -- 172.21.15.112:0/3719251211 >> v1:172.21.15.129:6789/0 conn(0x7f8dfc06a780 legacy=0x7f8dfc11b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933496884 0 0) 0x7f8dfc09bce0 con 0x7f8dfc06ab50 2024-09-18T12:01:47.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): my global_id is 134295 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ddc004f10 con 0x7f8dfc06ab50 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528406654 0 0) 0x7f8dfc09b200 con 0x7f8dfc06a780 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): my global_id is 144175 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dfc09bce0 con 0x7f8dfc06a780 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8df9d9b700 1 --1- 172.21.15.112:0/3719251211 >> v1:172.21.15.112:6789/0 conn(0x7f8dfc06ab50 0x7f8dfc11c040 :-1 s=OPENED pgs=686 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7f8df0002b70 mon_map magic: 0 v1 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3077601852 0 0) 0x7f8ddc004f10 con 0x7f8dfc06ab50 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dfc09b200 con 0x7f8dfc06ab50 2024-09-18T12:01:47.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8df0002b70 con 0x7f8dfc06ab50 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): dump: 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 464256776 0 0) 0x7f8dfc09bce0 con 0x7f8dfc06a780 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ddc004f10 con 0x7f8dfc06a780 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8df4001ef0 con 0x7f8dfc06a780 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient(hunting): dump: 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.303 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 354336299 0 0) 0x7f8ddc004f10 con 0x7f8dfc06a780 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 >> v1:172.21.15.112:6790/0 conn(0x7f8dfc135590 legacy=0x7f8dfc11c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 >> v1:172.21.15.112:6789/0 conn(0x7f8dfc06ab50 legacy=0x7f8dfc11c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 monclient: found mon.b 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc11d3d0 con 0x7f8dfc06a780 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient: _finish_auth 0 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4022239971 0 0) 0x7f8dfc09b200 con 0x7f8dfc06ab50 2024-09-18T12:01:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 5 monclient: authenticate success, global_id 144175 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 10 monclient: _renew_subs 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc11ef00 con 0x7f8dfc06a780 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 10 client.144175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 10 monclient: _renew_subs 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc11e140 con 0x7f8dfc06a780 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8df4002100 con 0x7f8dfc06a780 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 1 librados: init done 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8df4005100 con 0x7f8dfc06a780 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:47.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 monclient: dump: 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8df8d99700 4 set_mon_vals no callback set 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f8df4005260 con 0x7f8dfc06a780 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(688..688 src has 1..688) v4 ==== 5793+0+0 (unknown 1530601141 0 0) 0x7f8df4044210 con 0x7f8dfc06a780 2024-09-18T12:01:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 10 client.144175.objecter ms_dispatch 0x7f8dfc000fd0 osd_map(688..688 src has 1..688) v4 2024-09-18T12:01:47.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 3 client.144175.objecter handle_osd_map got epochs [688,688] > 0 2024-09-18T12:01:47.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 3 client.144175.objecter handle_osd_map decoding full epoch 688 2024-09-18T12:01:47.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:47.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:47.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:01:47.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8dea7fc700 20 client.144175.objecter dump_active .. 0 homeless 2024-09-18T12:01:47.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc11f370 con 0x7f8dfc06a780 2024-09-18T12:01:47.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.292+0000 7f8df9d9b700 10 monclient: get_auth_request con 0x7f8ddc0382b0 auth_method 0 2024-09-18T12:01:47.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.296+0000 7f8dea7fc700 10 client.144175.objecter ms_handle_connect 0x7f8ddc0382b0 2024-09-18T12:01:47.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.304+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f8df4005510 con 0x7f8dfc06a780 2024-09-18T12:01:47.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.304+0000 7f8dea7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:47.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.304+0000 7f8dea7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:47.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.532+0000 7f8e00e75700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:01:47.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.532+0000 7f8e00e75700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:01:47.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.532+0000 7f8e00e75700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:01:47.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.532+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8dfc05f2e0 con 0x7f8dfc06a780 2024-09-18T12:01:47.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.600+0000 7f8dea7fc700 1 -- 172.21.15.112:0/3719251211 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2627976716) 0x7f8df4014650 con 0x7f8dfc06a780 2024-09-18T12:01:47.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.600+0000 7f8dea7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:01:47.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.600+0000 7f8dea7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:47.604 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:01:47.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 10 librados: watch_flush enter 2024-09-18T12:01:47.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 10 librados: watch_flush exit 2024-09-18T12:01:47.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 20 client.144175.objecter shutdown clearing up homeless session... 2024-09-18T12:01:47.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 10 client.144175.objecter successfully canceled tick 2024-09-18T12:01:47.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 >> v1:172.21.15.129:6812/20806 conn(0x7f8ddc0382b0 legacy=0x7f8ddc03a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 10 monclient: shutdown 2024-09-18T12:01:47.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:47.609 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 >> v1:172.21.15.129:6789/0 conn(0x7f8dfc06a780 legacy=0x7f8dfc11b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 shutdown_connections 2024-09-18T12:01:47.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 >> 172.21.15.112:0/3719251211 conn(0x7f8dfc0680d0 msgr2=0x7f8dfc09c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:47.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 shutdown_connections 2024-09-18T12:01:47.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 1 -- 172.21.15.112:0/3719251211 wait complete. 2024-09-18T12:01:47.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.604+0000 7f8e00e75700 1 librados: shutdown 2024-09-18T12:01:47.622 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:01:47.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:01:47.623 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:01:47.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient: get_monmap_and_config 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient: build_initial_monmap 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient: monmap: 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:47.729051+0000 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:47.729051+0000 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:47.728 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 1 Processor -- start 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 1 -- start start 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient: init 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient(hunting): picked mon.noname-a con 0x7f693c0a1230 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient(hunting): picked mon.noname-c con 0x7f693c06e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient(hunting): picked mon.noname-b con 0x7f693c135300 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c0a1600 con 0x7f693c0a1230 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c06eda0 con 0x7f693c06e9d0 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c06b5a0 con 0x7f693c135300 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:47.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f6940a84700 10 monclient(hunting): authenticate will time out at 1879.569822s 2024-09-18T12:01:47.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f693ad9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f693c135300 0x7f693c1376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42594/0 (socket says 172.21.15.112:42594) 2024-09-18T12:01:47.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.724+0000 7f693ad9d700 1 -- 172.21.15.112:0/2921502280 learned_addr learned my addr 172.21.15.112:0/2921502280 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693ad9d700 1 -- 172.21.15.112:0/2921502280 >> v1:172.21.15.112:6790/0 conn(0x7f693c135300 legacy=0x7f693c1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6939d9b700 1 -- 172.21.15.112:0/2921502280 >> v1:172.21.15.129:6789/0 conn(0x7f693c06e9d0 legacy=0x7f693c06ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693a59c700 1 -- 172.21.15.112:0/2921502280 >> v1:172.21.15.112:6789/0 conn(0x7f693c0a1230 legacy=0x7f693c06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679920822 0 0) 0x7f693c0a1600 con 0x7f693c0a1230 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient(hunting): my global_id is 134298 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6924003570 con 0x7f693c0a1230 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799067087 0 0) 0x7f693c06eda0 con 0x7f693c06e9d0 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient(hunting): my global_id is 144178 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f693c0a1600 con 0x7f693c06e9d0 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2791662752 0 0) 0x7f693c0a1600 con 0x7f693c06e9d0 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 >> v1:172.21.15.112:6790/0 conn(0x7f693c135300 legacy=0x7f693c1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 >> v1:172.21.15.112:6789/0 conn(0x7f693c0a1230 legacy=0x7f693c06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: _finish_hunting 0 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 monclient: found mon.noname-c 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c09f9e0 con 0x7f693c06e9d0 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: _finish_auth 0 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:47.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 5 monclient: authenticate success, global_id 144178 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6930001a30 con 0x7f693c06e9d0 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: dump: 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.735 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.736 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:47.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6930001c10 con 0x7f693c06e9d0 2024-09-18T12:01:47.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:47.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:47.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 1 -- 172.21.15.112:0/2921502280 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f69300020a0 con 0x7f693c06e9d0 2024-09-18T12:01:47.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 4 set_mon_vals no callback set 2024-09-18T12:01:47.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:47.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f693959a700 10 monclient: dump: 2024-09-18T12:01:47.737 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.738 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 10 monclient: shutdown 2024-09-18T12:01:47.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:47.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 1 -- 172.21.15.112:0/2921502280 >> v1:172.21.15.129:6789/0 conn(0x7f693c06e9d0 legacy=0x7f693c06ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 1 -- 172.21.15.112:0/2921502280 shutdown_connections 2024-09-18T12:01:47.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 1 -- 172.21.15.112:0/2921502280 >> 172.21.15.112:0/2921502280 conn(0x7f693c0680d0 msgr2=0x7f693c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:47.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 1 -- 172.21.15.112:0/2921502280 shutdown_connections 2024-09-18T12:01:47.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 1 -- 172.21.15.112:0/2921502280 wait complete. 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 10 monclient: build_initial_monmap 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.728+0000 7f6940a84700 10 monclient: monmap: 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 librados: starting msgr at 2024-09-18T12:01:47.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 librados: starting objecter 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 Processor -- start 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 -- start start 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 librados: setting wanted keys 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 librados: calling monclient init 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 10 monclient: init 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 10 monclient(hunting): picked mon.noname-c con 0x7f693c06e9d0 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 10 monclient(hunting): picked mon.noname-a con 0x7f693c0a1230 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 10 monclient(hunting): picked mon.noname-b con 0x7f693c135300 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c12f170 con 0x7f693c0a1230 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c134110 con 0x7f693c135300 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c118810 con 0x7f693c06e9d0 2024-09-18T12:01:47.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6940a84700 10 monclient(hunting): authenticate will time out at 1879.577192s 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6939d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f693c0a1230 0x7f693c146a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56862/0 (socket says 172.21.15.112:56862) 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6939d9b700 1 -- 172.21.15.112:0/84379454 learned_addr learned my addr 172.21.15.112:0/84379454 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f6939d9b700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6789/0 conn(0x7f693c0a1230 legacy=0x7f693c146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 client.?.objecter ms_handle_connect 0x7f693c0a1230 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 client.?.objecter ms_handle_connect 0x7f693c06e9d0 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f693a59c700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6790/0 conn(0x7f693c06e9d0 legacy=0x7f693c12e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 client.?.objecter ms_handle_connect 0x7f693c135300 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f693ad9d700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.129:6789/0 conn(0x7f693c135300 legacy=0x7f693c14a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674040381 0 0) 0x7f693c12f170 con 0x7f693c0a1230 2024-09-18T12:01:47.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 monclient(hunting): my global_id is 134301 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c004f10 con 0x7f693c0a1230 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916263762 0 0) 0x7f693c118810 con 0x7f693c06e9d0 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 10 monclient(hunting): my global_id is 154100 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f693c12f170 con 0x7f693c06e9d0 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.732+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782335700 0 0) 0x7f693c134110 con 0x7f693c135300 2024-09-18T12:01:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): my global_id is 144181 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f693c118810 con 0x7f693c135300 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 170685429 0 0) 0x7f691c004f10 con 0x7f693c0a1230 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f693c134110 con 0x7f693c0a1230 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6930001850 con 0x7f693c0a1230 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): dump: 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1154262827 0 0) 0x7f693c12f170 con 0x7f693c06e9d0 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f691c004f10 con 0x7f693c06e9d0 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f692c001bd0 con 0x7f693c06e9d0 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): dump: 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 674837121 0 0) 0x7f693c118810 con 0x7f693c135300 2024-09-18T12:01:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f693c12f170 con 0x7f693c135300 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6934001e00 con 0x7f693c135300 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient(hunting): dump: 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.753 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1284584188 0 0) 0x7f693c134110 con 0x7f693c0a1230 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6790/0 conn(0x7f693c06e9d0 legacy=0x7f693c12e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.129:6789/0 conn(0x7f693c135300 legacy=0x7f693c14a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 monclient: found mon.a 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c118af0 con 0x7f693c0a1230 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient: _finish_auth 0 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 5 monclient: authenticate success, global_id 134301 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 10 monclient: _renew_subs 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c1421f0 con 0x7f693c0a1230 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 10 client.134301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 10 monclient: _renew_subs 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c12ee30 con 0x7f693c0a1230 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 1 librados: init done 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6930002680 con 0x7f693c0a1230 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6930002a10 con 0x7f693c0a1230 2024-09-18T12:01:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6938d99700 4 set_mon_vals no callback set 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 monclient: dump: 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.756 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6920006590 con 0x7f6920002840 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f6930002b70 con 0x7f693c0a1230 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(689..689 src has 1..689) v4 ==== 5806+0+0 (unknown 2101935416 0 0) 0x7f6930044260 con 0x7f693c0a1230 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 10 client.134301.objecter ms_dispatch 0x7f693c000fd0 osd_map(689..689 src has 1..689) v4 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 3 client.134301.objecter handle_osd_map got epochs [689,689] > 0 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 3 client.134301.objecter handle_osd_map decoding full epoch 689 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f692a7fc700 20 client.134301.objecter dump_active .. 0 homeless 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.736+0000 7f693ad9d700 10 monclient: get_auth_request con 0x7f691c038650 auth_method 0 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f693a59c700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6789/0 conn(0x7f6920002840 legacy=0x7f6920004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 client.134301.objecter ms_handle_connect 0x7f6920002840 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 client.134301.objecter resend_mon_ops 2024-09-18T12:01:47.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161240868 0 0) 0x7f6920006590 con 0x7f6920002840 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 monclient(hunting): my global_id is 134304 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f693c134110 con 0x7f6920002840 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 991688374 0 0) 0x7f693c134110 con 0x7f6920002840 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6920006590 con 0x7f6920002840 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f692c0023d0 con 0x7f6920002840 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 monclient: dump: 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.758 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2021449310 0 0) 0x7f6920006590 con 0x7f6920002840 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f692000a140 con 0x7f6920002840 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 client.134301.objecter ms_handle_connect 0x7f691c038650 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f692c005810 con 0x7f6920002840 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:47.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6789/0 conn(0x7f6920002840 legacy=0x7f6920004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.740+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6789/0 conn(0x7f6920002840 legacy=0x7f6920004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:47.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f6940a84700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:01:47.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6920009f10 con 0x7f6920006900 2024-09-18T12:01:47.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f692a7fc700 10 client.134301.objecter ms_handle_connect 0x7f6920006900 2024-09-18T12:01:47.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f692a7fc700 10 client.134301.objecter resend_mon_ops 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f693a59c700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6789/0 conn(0x7f6920006900 legacy=0x7f6920006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425271609 0 0) 0x7f6920009f10 con 0x7f6920006900 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f692a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f692a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f692a7fc700 10 monclient(hunting): my global_id is 134307 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6920006590 con 0x7f6920006900 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.748+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4022240517 0 0) 0x7f6920006590 con 0x7f6920006900 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6920009f10 con 0x7f6920006900 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f692c002a40 con 0x7f6920006900 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 10 monclient: dump: 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1370692950 0 0) 0x7f6920009f10 con 0x7f6920006900 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6920008760 con 0x7f6920006900 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4173 (unknown 0 0 3093827663) 0x7f692c002bc0 con 0x7f6920006900 2024-09-18T12:01:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6789/0 conn(0x7f6920006900 legacy=0x7f6920006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f692a7fc700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6789/0 conn(0x7f6920006900 legacy=0x7f6920006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 76, 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:01:47.764 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:47.765 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:47.766 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:47.767 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:47.768 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.769 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:47.770 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:01:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:01:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:01:47.773 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.774 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 5 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.775 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:01:47.776 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 10 librados: watch_flush enter 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 10 librados: watch_flush exit 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 20 client.134301.objecter shutdown clearing up homeless session... 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 10 client.134301.objecter successfully canceled tick 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.129:6812/20806 conn(0x7f691c038650 legacy=0x7f691c03ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 10 monclient: shutdown 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 >> v1:172.21.15.112:6789/0 conn(0x7f693c0a1230 legacy=0x7f693c146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f693ad9d700 1 -- 172.21.15.112:0/84379454 reap_dead start 2024-09-18T12:01:47.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f693ad9d700 1 -- 172.21.15.112:0/84379454 reap_dead start 2024-09-18T12:01:47.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 shutdown_connections 2024-09-18T12:01:47.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 >> 172.21.15.112:0/84379454 conn(0x7f693c0680d0 msgr2=0x7f693c09ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:47.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 shutdown_connections 2024-09-18T12:01:47.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 1 -- 172.21.15.112:0/84379454 wait complete. 2024-09-18T12:01:47.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.752+0000 7f6940a84700 1 librados: shutdown 2024-09-18T12:01:47.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:01:47.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:01:47.779 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.884+0000 7fca2759e700 10 monclient: get_monmap_and_config 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.884+0000 7fca2759e700 10 monclient: build_initial_monmap 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.884+0000 7fca2759e700 10 monclient: monmap: 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:47.891265+0000 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:47.891265+0000 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:47.890 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:47.891 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:47.891 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.884+0000 7fca2759e700 1 Processor -- start 2024-09-18T12:01:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.884+0000 7fca2759e700 1 -- start start 2024-09-18T12:01:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.884+0000 7fca2759e700 10 monclient: init 2024-09-18T12:01:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:01:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 10 monclient(hunting): picked mon.noname-a con 0x7fca280a15f0 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:47.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 10 monclient(hunting): picked mon.noname-c con 0x7fca280a19c0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 10 monclient(hunting): picked mon.noname-b con 0x7fca28138530 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca280a3250 con 0x7fca280a15f0 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca28064370 con 0x7fca280a19c0 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2806a5a0 con 0x7fca28138530 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 10 monclient(hunting): authenticate will time out at 1879.732448s 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca26d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fca28138530 0x7fca2813a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42620/0 (socket says 172.21.15.112:42620) 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fca280a15f0 0x7fca2806c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56892/0 (socket says 172.21.15.112:56892) 2024-09-18T12:01:47.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca26d9d700 1 -- 172.21.15.112:0/2873738159 learned_addr learned my addr 172.21.15.112:0/2873738159 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:47.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2659c700 1 -- 172.21.15.112:0/2873738159 >> v1:172.21.15.112:6789/0 conn(0x7fca280a15f0 legacy=0x7fca2806c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca26d9d700 1 -- 172.21.15.112:0/2873738159 >> v1:172.21.15.112:6790/0 conn(0x7fca28138530 legacy=0x7fca2813a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca25d9b700 1 -- 172.21.15.112:0/2873738159 >> v1:172.21.15.129:6789/0 conn(0x7fca280a19c0 legacy=0x7fca280a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255202462 0 0) 0x7fca280a3250 con 0x7fca280a15f0 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): my global_id is 134310 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca10003570 con 0x7fca280a15f0 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240385367 0 0) 0x7fca2806a5a0 con 0x7fca28138530 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): my global_id is 154103 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca280a3250 con 0x7fca28138530 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298149716 0 0) 0x7fca28064370 con 0x7fca280a19c0 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient(hunting): my global_id is 144184 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca2806a5a0 con 0x7fca280a19c0 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2314512530 0 0) 0x7fca10003570 con 0x7fca280a15f0 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 >> v1:172.21.15.112:6790/0 conn(0x7fca28138530 legacy=0x7fca2813a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 >> v1:172.21.15.129:6789/0 conn(0x7fca280a19c0 legacy=0x7fca280a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient: _finish_hunting 0 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 monclient: found mon.noname-a 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2806b590 con 0x7fca280a15f0 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient: _finish_auth 0 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 5 monclient: authenticate success, global_id 134310 2024-09-18T12:01:47.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fca1c001a00 con 0x7fca280a15f0 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2559a700 10 monclient: dump: 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.899 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.888+0000 7fca2759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca1c001be0 con 0x7fca280a15f0 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2559a700 1 -- 172.21.15.112:0/2873738159 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fca1c0049d0 con 0x7fca280a15f0 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 4 set_mon_vals no callback set 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2559a700 10 monclient: dump: 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.900 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient: shutdown 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- 172.21.15.112:0/2873738159 >> v1:172.21.15.112:6789/0 conn(0x7fca280a15f0 legacy=0x7fca2806c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- 172.21.15.112:0/2873738159 shutdown_connections 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- 172.21.15.112:0/2873738159 >> 172.21.15.112:0/2873738159 conn(0x7fca280a0da0 msgr2=0x7fca28067680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- 172.21.15.112:0/2873738159 shutdown_connections 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- 172.21.15.112:0/2873738159 wait complete. 2024-09-18T12:01:47.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient: build_initial_monmap 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient: monmap: 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 librados: starting msgr at 2024-09-18T12:01:47.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 librados: starting objecter 2024-09-18T12:01:47.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 Processor -- start 2024-09-18T12:01:47.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- start start 2024-09-18T12:01:47.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 librados: setting wanted keys 2024-09-18T12:01:47.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 librados: calling monclient init 2024-09-18T12:01:47.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient: init 2024-09-18T12:01:47.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient(hunting): picked mon.noname-c con 0x7fca280a15f0 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient(hunting): picked mon.noname-b con 0x7fca280a19c0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient(hunting): picked mon.noname-a con 0x7fca28138530 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2809fb30 con 0x7fca28138530 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca281361a0 con 0x7fca280a19c0 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2806f000 con 0x7fca280a15f0 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2759e700 10 monclient(hunting): authenticate will time out at 1879.739178s 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2659c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fca280a15f0 0x7fca28141400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42636/0 (socket says 172.21.15.112:42636) 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.892+0000 7fca2659c700 1 -- 172.21.15.112:0/1101520257 learned_addr learned my addr 172.21.15.112:0/1101520257 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2659c700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6790/0 conn(0x7fca280a15f0 legacy=0x7fca28141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 client.?.objecter ms_handle_connect 0x7fca280a15f0 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 client.?.objecter ms_handle_connect 0x7fca28138530 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca26d9d700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6789/0 conn(0x7fca28138530 legacy=0x7fca28149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca25d9b700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.129:6789/0 conn(0x7fca280a19c0 legacy=0x7fca28132390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025574930 0 0) 0x7fca2806f000 con 0x7fca280a15f0 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): my global_id is 154106 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca08004f10 con 0x7fca280a15f0 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 client.?.objecter ms_handle_connect 0x7fca280a19c0 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:47.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782031180 0 0) 0x7fca2809fb30 con 0x7fca28138530 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): my global_id is 134313 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca2806f000 con 0x7fca28138530 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813895694 0 0) 0x7fca281361a0 con 0x7fca280a19c0 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): my global_id is 144187 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca2809fb30 con 0x7fca280a19c0 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 171419182 0 0) 0x7fca08004f10 con 0x7fca280a15f0 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca281361a0 con 0x7fca280a15f0 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fca1c001820 con 0x7fca280a15f0 2024-09-18T12:01:47.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): dump: 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:47.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1232804586 0 0) 0x7fca2806f000 con 0x7fca28138530 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca08004f10 con 0x7fca28138530 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fca20004bb0 con 0x7fca28138530 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): dump: 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.914 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2751762191 0 0) 0x7fca2809fb30 con 0x7fca280a19c0 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca2806f000 con 0x7fca280a19c0 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fca18001c00 con 0x7fca280a19c0 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient(hunting): dump: 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 949186477 0 0) 0x7fca281361a0 con 0x7fca280a15f0 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.129:6789/0 conn(0x7fca280a19c0 legacy=0x7fca28132390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6789/0 conn(0x7fca28138530 legacy=0x7fca28149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient: _finish_hunting 0 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 monclient: found mon.c 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca28136860 con 0x7fca280a15f0 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient: _finish_auth 0 2024-09-18T12:01:47.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 5 monclient: authenticate success, global_id 154106 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 10 monclient: _renew_subs 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca28142240 con 0x7fca280a15f0 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 10 client.154106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 10 monclient: _renew_subs 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca28132aa0 con 0x7fca280a15f0 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca2759e700 1 librados: init done 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca1c005030 con 0x7fca280a15f0 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fca1c005340 con 0x7fca280a15f0 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca24d99700 4 set_mon_vals no callback set 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.896+0000 7fca167fc700 10 monclient: dump: 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca2759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:47.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fca0c006590 con 0x7fca0c002840 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fca1c00fbc0 con 0x7fca280a15f0 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(689..689 src has 1..689) v4 ==== 5806+0+0 (unknown 2101935416 0 0) 0x7fca1c0441e0 con 0x7fca280a15f0 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 client.154106.objecter ms_dispatch 0x7fca28000fd0 osd_map(689..689 src has 1..689) v4 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 3 client.154106.objecter handle_osd_map got epochs [689,689] > 0 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 3 client.154106.objecter handle_osd_map decoding full epoch 689 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 20 client.154106.objecter dump_active .. 0 homeless 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca26d9d700 10 monclient: get_auth_request con 0x7fca08038750 auth_method 0 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 client.154106.objecter ms_handle_connect 0x7fca0c002840 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca25d9b700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6790/0 conn(0x7fca0c002840 legacy=0x7fca0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 client.154106.objecter resend_mon_ops 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201975908 0 0) 0x7fca0c006590 con 0x7fca0c002840 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 monclient(hunting): my global_id is 154109 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca281361a0 con 0x7fca0c002840 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1038553650 0 0) 0x7fca281361a0 con 0x7fca0c002840 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c006590 con 0x7fca0c002840 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fca180023d0 con 0x7fca0c002840 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.900+0000 7fca167fc700 10 monclient: dump: 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.904+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1636715099 0 0) 0x7fca0c006590 con 0x7fca0c002840 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.904+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fca0c00a140 con 0x7fca0c002840 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.904+0000 7fca167fc700 10 client.154106.objecter ms_handle_connect 0x7fca08038750 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.904+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fca18005810 con 0x7fca0c002840 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.904+0000 7fca167fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.904+0000 7fca167fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.904+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6790/0 conn(0x7fca0c002840 legacy=0x7fca0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.904+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6790/0 conn(0x7fca0c002840 legacy=0x7fca0c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.912+0000 7fca2759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.912+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fca0c009f10 con 0x7fca0c006900 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca25d9b700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6790/0 conn(0x7fca0c006900 legacy=0x7fca0c006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 client.154106.objecter ms_handle_connect 0x7fca0c006900 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 client.154106.objecter resend_mon_ops 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413988944 0 0) 0x7fca0c009f10 con 0x7fca0c006900 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 monclient(hunting): my global_id is 154112 2024-09-18T12:01:47.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c006590 con 0x7fca0c006900 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2901462902 0 0) 0x7fca0c006590 con 0x7fca0c006900 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c009f10 con 0x7fca0c006900 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fca18002a40 con 0x7fca0c006900 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 monclient: dump: 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:47.924 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 246645831 0 0) 0x7fca0c009f10 con 0x7fca0c006900 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fca0c008760 con 0x7fca0c006900 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 1806554988) 0x7fca18002ca0 con 0x7fca0c006900 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6790/0 conn(0x7fca0c006900 legacy=0x7fca0c006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca167fc700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6790/0 conn(0x7fca0c006900 legacy=0x7fca0c006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:47.925 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 76, 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:01:47.926 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:01:47.928 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:47.929 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:01:47.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:01:47.931 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:47.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:47.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.934 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:47.935 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T12:01:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:47.936 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:01:47.936 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca2759e700 10 librados: watch_flush enter 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca2759e700 10 librados: watch_flush exit 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca2759e700 20 client.154106.objecter shutdown clearing up homeless session... 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca2759e700 10 client.154106.objecter successfully canceled tick 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.916+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.129:6812/20806 conn(0x7fca08038750 legacy=0x7fca0803ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca2759e700 10 monclient: shutdown 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca2759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 >> v1:172.21.15.112:6790/0 conn(0x7fca280a15f0 legacy=0x7fca28141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca26d9d700 1 -- 172.21.15.112:0/1101520257 reap_dead start 2024-09-18T12:01:47.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca26d9d700 1 -- 172.21.15.112:0/1101520257 reap_dead start 2024-09-18T12:01:47.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 shutdown_connections 2024-09-18T12:01:47.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 >> 172.21.15.112:0/1101520257 conn(0x7fca280a0da0 msgr2=0x7fca2806e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:47.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 shutdown_connections 2024-09-18T12:01:47.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca2759e700 1 -- 172.21.15.112:0/1101520257 wait complete. 2024-09-18T12:01:47.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:47.920+0000 7fca2759e700 1 librados: shutdown 2024-09-18T12:01:47.939 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:01:47.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:01:47.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:01:48.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.048+0000 7f34cf59e700 10 monclient: get_monmap_and_config 2024-09-18T12:01:48.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.048+0000 7f34cf59e700 10 monclient: build_initial_monmap 2024-09-18T12:01:48.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.048+0000 7f34cf59e700 10 monclient: monmap: 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:48.055474+0000 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:48.055474+0000 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:48.056 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.048+0000 7f34cf59e700 1 Processor -- start 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.048+0000 7f34cf59e700 1 -- start start 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.048+0000 7f34cf59e700 10 monclient: init 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f34d00a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f34d00a0750 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f34d01385b0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d0097270 con 0x7f34d00a0750 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d00a3170 con 0x7f34d01385b0 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d0096cb0 con 0x7f34d00a0380 2024-09-18T12:01:48.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:48.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 10 monclient(hunting): authenticate will time out at 1879.896498s 2024-09-18T12:01:48.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cdd9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f34d00a0750 0x7f34d00a0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56912/0 (socket says 172.21.15.112:56912) 2024-09-18T12:01:48.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cdd9b700 1 -- 172.21.15.112:0/1533674121 learned_addr learned my addr 172.21.15.112:0/1533674121 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:48.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cdd9b700 1 -- 172.21.15.112:0/1533674121 >> v1:172.21.15.112:6789/0 conn(0x7f34d00a0750 legacy=0x7f34d00a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34ce59c700 1 -- 172.21.15.112:0/1533674121 >> v1:172.21.15.112:6790/0 conn(0x7f34d00a0380 legacy=0x7f34d0097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117020474 0 0) 0x7f34d0097270 con 0x7f34d00a0750 2024-09-18T12:01:48.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34ced9d700 1 -- 172.21.15.112:0/1533674121 >> v1:172.21.15.129:6789/0 conn(0x7f34d01385b0 legacy=0x7f34d013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient(hunting): my global_id is 134316 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b8003570 con 0x7f34d00a0750 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461423614 0 0) 0x7f34d0096cb0 con 0x7f34d00a0380 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient(hunting): my global_id is 154115 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d0097270 con 0x7f34d00a0380 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2483034325 0 0) 0x7f34b8003570 con 0x7f34d00a0750 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 >> v1:172.21.15.112:6790/0 conn(0x7f34d00a0380 legacy=0x7f34d0097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 >> v1:172.21.15.129:6789/0 conn(0x7f34d01385b0 legacy=0x7f34d013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient: _finish_hunting 0 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 monclient: found mon.noname-a 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00a1420 con 0x7f34d00a0750 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient: _finish_auth 0 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 5 monclient: authenticate success, global_id 134316 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f34c0001d00 con 0x7f34d00a0750 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cd59a700 10 monclient: dump: 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.052+0000 7f34cf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34c0001e60 con 0x7f34d00a0750 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cd59a700 1 -- 172.21.15.112:0/1533674121 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f34c0004e10 con 0x7f34d00a0750 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 4 set_mon_vals no callback set 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cd59a700 10 monclient: dump: 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient: shutdown 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- 172.21.15.112:0/1533674121 >> v1:172.21.15.112:6789/0 conn(0x7f34d00a0750 legacy=0x7f34d00a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- 172.21.15.112:0/1533674121 shutdown_connections 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- 172.21.15.112:0/1533674121 >> 172.21.15.112:0/1533674121 conn(0x7f34d00a5990 msgr2=0x7f34d00a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- 172.21.15.112:0/1533674121 shutdown_connections 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- 172.21.15.112:0/1533674121 wait complete. 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient: build_initial_monmap 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient: monmap: 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 librados: starting msgr at 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 librados: starting objecter 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 Processor -- start 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- start start 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 librados: setting wanted keys 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 librados: calling monclient init 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient: init 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f34d00a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f34d00a0750 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f34d01385b0 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d00a2010 con 0x7f34d01385b0 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d009efc0 con 0x7f34d00a0750 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d009c0d0 con 0x7f34d00a0380 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34cf59e700 10 monclient(hunting): authenticate will time out at 1879.902589s 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34ce59c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f34d00a0380 0x7f34d0097e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42672/0 (socket says 172.21.15.112:42672) 2024-09-18T12:01:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.056+0000 7f34ce59c700 1 -- 172.21.15.112:0/1684349070 learned_addr learned my addr 172.21.15.112:0/1684349070 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:48.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34ce59c700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.112:6790/0 conn(0x7f34d00a0380 legacy=0x7f34d0097e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 client.?.objecter ms_handle_connect 0x7f34d00a0380 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34ced9d700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.112:6789/0 conn(0x7f34d01385b0 legacy=0x7f34d014e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 client.?.objecter ms_handle_connect 0x7f34d01385b0 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 client.?.objecter ms_handle_connect 0x7f34d00a0750 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cdd9b700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34d00a0750 legacy=0x7f34d0098540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165300739 0 0) 0x7f34d00a2010 con 0x7f34d01385b0 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): my global_id is 134319 2024-09-18T12:01:48.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b0004f10 con 0x7f34d01385b0 2024-09-18T12:01:48.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483721580 0 0) 0x7f34d009c0d0 con 0x7f34d00a0380 2024-09-18T12:01:48.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): my global_id is 154118 2024-09-18T12:01:48.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d00a2010 con 0x7f34d00a0380 2024-09-18T12:01:48.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795556675 0 0) 0x7f34d009efc0 con 0x7f34d00a0750 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): my global_id is 144193 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d009c0d0 con 0x7f34d00a0750 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 255385662 0 0) 0x7f34b0004f10 con 0x7f34d01385b0 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d009efc0 con 0x7f34d01385b0 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f34c8001bc0 con 0x7f34d01385b0 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): dump: 2024-09-18T12:01:48.081 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1551330888 0 0) 0x7f34d00a2010 con 0x7f34d00a0380 2024-09-18T12:01:48.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b0004f10 con 0x7f34d00a0380 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f34c4001d00 con 0x7f34d00a0380 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): dump: 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.083 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3203956101 0 0) 0x7f34d009c0d0 con 0x7f34d00a0750 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d00a2010 con 0x7f34d00a0750 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f34c0004bb0 con 0x7f34d00a0750 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient(hunting): dump: 2024-09-18T12:01:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1042315892 0 0) 0x7f34d009efc0 con 0x7f34d01385b0 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.112:6790/0 conn(0x7f34d00a0380 legacy=0x7f34d0097e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34d00a0750 legacy=0x7f34d0098540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 monclient: found mon.a 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d0098c50 con 0x7f34d01385b0 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient: _finish_auth 0 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 5 monclient: authenticate success, global_id 134319 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 10 monclient: _renew_subs 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d0099df0 con 0x7f34d01385b0 2024-09-18T12:01:48.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 10 client.134319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:48.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 10 monclient: _renew_subs 2024-09-18T12:01:48.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d009d8c0 con 0x7f34d01385b0 2024-09-18T12:01:48.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34cf59e700 1 librados: init done 2024-09-18T12:01:48.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34c80021b0 con 0x7f34d01385b0 2024-09-18T12:01:48.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:48.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f34c8002540 con 0x7f34d01385b0 2024-09-18T12:01:48.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:48.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34be7fc700 10 monclient: dump: 2024-09-18T12:01:48.088 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.088 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.088 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.088 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.060+0000 7f34ccd99700 4 set_mon_vals no callback set 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f34c80026a0 con 0x7f34d01385b0 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34cf59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f34b4006590 con 0x7f34b4002840 2024-09-18T12:01:48.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(689..689 src has 1..689) v4 ==== 5806+0+0 (unknown 2101935416 0 0) 0x7f34c8043e90 con 0x7f34d01385b0 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 10 client.134319.objecter ms_dispatch 0x7f34d0000fd0 osd_map(689..689 src has 1..689) v4 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 3 client.134319.objecter handle_osd_map got epochs [689,689] > 0 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 3 client.134319.objecter handle_osd_map decoding full epoch 689 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 20 client.134319.objecter dump_active .. 0 homeless 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34cdd9b700 10 monclient: get_auth_request con 0x7f34b0038560 auth_method 0 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 10 client.134319.objecter ms_handle_connect 0x7f34b4002840 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 10 client.134319.objecter resend_mon_ops 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34ce59c700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b4002840 legacy=0x7f34b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328146775 0 0) 0x7f34b4006590 con 0x7f34b4002840 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 10 monclient(hunting): my global_id is 144196 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d009efc0 con 0x7f34b4002840 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 10 client.134319.objecter ms_handle_connect 0x7f34b0038560 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 878003067 0 0) 0x7f34d009efc0 con 0x7f34b4002840 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34ce59c700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b4002840 legacy=0x7f34b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34ce59c700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b4002840 legacy=0x7f34b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34ce59c700 1 --1- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b4002840 0x7f34b4004cd0 :-1 s=OPENED pgs=9253 cs=1 l=1).handle_message read tag failed 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.064+0000 7f34ce59c700 1 --1- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b4002840 0x7f34b4004cd0 :-1 s=OPENED pgs=9253 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b4006590 con 0x7f34b4002840 2024-09-18T12:01:48.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b4002840 legacy=0x7f34b4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b4002840 legacy=0x7f34b4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f34d009efc0 con 0x7f34b0042d10 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 10 client.134319.objecter ms_handle_connect 0x7f34b0042d10 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34ce59c700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b0042d10 legacy=0x7f34b0043110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 10 client.134319.objecter resend_mon_ops 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364885211 0 0) 0x7f34d009efc0 con 0x7f34b0042d10 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 10 monclient(hunting): my global_id is 144199 2024-09-18T12:01:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b4006590 con 0x7f34b0042d10 2024-09-18T12:01:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1824398669 0 0) 0x7f34b4006590 con 0x7f34b0042d10 2024-09-18T12:01:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d009efc0 con 0x7f34b0042d10 2024-09-18T12:01:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f34c4002a70 con 0x7f34b0042d10 2024-09-18T12:01:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 10 monclient: dump: 2024-09-18T12:01:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 834076105 0 0) 0x7f34d009efc0 con 0x7f34b0042d10 2024-09-18T12:01:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.068+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34b00438b0 con 0x7f34b0042d10 2024-09-18T12:01:48.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.072+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f34c4005810 con 0x7f34b0042d10 2024-09-18T12:01:48.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.072+0000 7f34be7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:48.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.072+0000 7f34be7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:48.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.072+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b0042d10 legacy=0x7f34b0043110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.072+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b0042d10 legacy=0x7f34b0043110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:48.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.080+0000 7f34cf59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:01:48.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.080+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f34b4009630 con 0x7f34b40090f0 2024-09-18T12:01:48.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34ce59c700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b40090f0 legacy=0x7f34b4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 client.134319.objecter ms_handle_connect 0x7f34b40090f0 2024-09-18T12:01:48.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 client.134319.objecter resend_mon_ops 2024-09-18T12:01:48.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931305264 0 0) 0x7f34b4009630 con 0x7f34b40090f0 2024-09-18T12:01:48.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 monclient(hunting): my global_id is 144202 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d009efc0 con 0x7f34b40090f0 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2930909749 0 0) 0x7f34d009efc0 con 0x7f34b40090f0 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b4009630 con 0x7f34b40090f0 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f34c4005c90 con 0x7f34b40090f0 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 monclient: dump: 2024-09-18T12:01:48.098 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2074135704 0 0) 0x7f34b4009630 con 0x7f34b40090f0 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f34b4009880 con 0x7f34b40090f0 2024-09-18T12:01:48.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 4154332413) 0x7f34c4006fb0 con 0x7f34b40090f0 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b40090f0 legacy=0x7f34b4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34be7fc700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6789/0 conn(0x7f34b40090f0 legacy=0x7f34b4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34ced9d700 1 -- 172.21.15.112:0/1684349070 reap_dead start 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:01:48.100 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 76, 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:01:48.101 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:48.102 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:48.103 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:48.104 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:48.105 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:01:48.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.107 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:48.108 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:01:48.109 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:01:48.110 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.111 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:48.112 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:01:48.113 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:01:48.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 10 librados: watch_flush enter 2024-09-18T12:01:48.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 10 librados: watch_flush exit 2024-09-18T12:01:48.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 20 client.134319.objecter shutdown clearing up homeless session... 2024-09-18T12:01:48.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 10 client.134319.objecter successfully canceled tick 2024-09-18T12:01:48.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.129:6812/20806 conn(0x7f34b0038560 legacy=0x7f34b003aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 10 monclient: shutdown 2024-09-18T12:01:48.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:48.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 >> v1:172.21.15.112:6789/0 conn(0x7f34d01385b0 legacy=0x7f34d014e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 shutdown_connections 2024-09-18T12:01:48.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 >> 172.21.15.112:0/1684349070 conn(0x7f34d00a5990 msgr2=0x7f34d0138980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:48.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 shutdown_connections 2024-09-18T12:01:48.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.084+0000 7f34cf59e700 1 -- 172.21.15.112:0/1684349070 wait complete. 2024-09-18T12:01:48.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.088+0000 7f34cf59e700 1 librados: shutdown 2024-09-18T12:01:48.116 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:01:48.116 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:01:48.117 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:01:48.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient: get_monmap_and_config 2024-09-18T12:01:48.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient: build_initial_monmap 2024-09-18T12:01:48.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient: monmap: 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:48.229799+0000 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:48.229799+0000 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:48.229 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 1 Processor -- start 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 1 -- start start 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient: init 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient(hunting): picked mon.noname-c con 0x7fe7e80a1230 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient(hunting): picked mon.noname-a con 0x7fe7e806e9d0 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient(hunting): picked mon.noname-b con 0x7fe7e8135300 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80a1600 con 0x7fe7e806e9d0 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e806eda0 con 0x7fe7e80a1230 2024-09-18T12:01:48.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e806b5a0 con 0x7fe7e8135300 2024-09-18T12:01:48.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:48.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7ed342700 10 monclient(hunting): authenticate will time out at 1880.071349s 2024-09-18T12:01:48.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7e77fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe7e8135300 0x7fe7e81376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42680/0 (socket says 172.21.15.112:42680) 2024-09-18T12:01:48.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7e77fe700 1 -- 172.21.15.112:0/2874759601 learned_addr learned my addr 172.21.15.112:0/2874759601 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:48.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.224+0000 7fe7de7fc700 1 --1- 172.21.15.112:0/2874759601 >> v1:172.21.15.112:6789/0 conn(0x7fe7e806e9d0 0x7fe7e806ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56922/0 (socket says 172.21.15.112:56922) 2024-09-18T12:01:48.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e77fe700 1 -- 172.21.15.112:0/2874759601 >> v1:172.21.15.112:6790/0 conn(0x7fe7e8135300 legacy=0x7fe7e81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7de7fc700 1 -- 172.21.15.112:0/2874759601 >> v1:172.21.15.112:6789/0 conn(0x7fe7e806e9d0 legacy=0x7fe7e806ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e6ffd700 1 -- 172.21.15.112:0/2874759601 >> v1:172.21.15.129:6789/0 conn(0x7fe7e80a1230 legacy=0x7fe7e806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437154876 0 0) 0x7fe7e806b5a0 con 0x7fe7e8135300 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): my global_id is 154121 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d4003570 con 0x7fe7e8135300 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996515403 0 0) 0x7fe7e80a1600 con 0x7fe7e806e9d0 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): my global_id is 134322 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e806b5a0 con 0x7fe7e806e9d0 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931747804 0 0) 0x7fe7e806eda0 con 0x7fe7e80a1230 2024-09-18T12:01:48.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient(hunting): my global_id is 144205 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e80a1600 con 0x7fe7e80a1230 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686558954 0 0) 0x7fe7d4003570 con 0x7fe7e8135300 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 >> v1:172.21.15.129:6789/0 conn(0x7fe7e80a1230 legacy=0x7fe7e806a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 >> v1:172.21.15.112:6789/0 conn(0x7fe7e806e9d0 legacy=0x7fe7e806ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: _finish_hunting 0 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 monclient: found mon.noname-b 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e809f9e0 con 0x7fe7e8135300 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: _finish_auth 0 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7e0001cc0 con 0x7fe7e8135300 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: dump: 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 5 monclient: authenticate success, global_id 154121 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7e0001f50 con 0x7fe7e8135300 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 1 -- 172.21.15.112:0/2874759601 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7e0004e10 con 0x7fe7e8135300 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 4 set_mon_vals no callback set 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7e67fc700 10 monclient: dump: 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.238 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 10 monclient: shutdown 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 1 -- 172.21.15.112:0/2874759601 >> v1:172.21.15.112:6790/0 conn(0x7fe7e8135300 legacy=0x7fe7e81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 1 -- 172.21.15.112:0/2874759601 shutdown_connections 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 1 -- 172.21.15.112:0/2874759601 >> 172.21.15.112:0/2874759601 conn(0x7fe7e80680d0 msgr2=0x7fe7e80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 1 -- 172.21.15.112:0/2874759601 shutdown_connections 2024-09-18T12:01:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.228+0000 7fe7ed342700 1 -- 172.21.15.112:0/2874759601 wait complete. 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient: build_initial_monmap 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient: monmap: 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:48.245 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 librados: starting msgr at 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 librados: starting objecter 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 Processor -- start 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 -- start start 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 librados: setting wanted keys 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 librados: calling monclient init 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient: init 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient(hunting): picked mon.noname-c con 0x7fe7e806e9d0 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient(hunting): picked mon.noname-b con 0x7fe7e80a1230 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:48.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient(hunting): picked mon.noname-a con 0x7fe7e8135300 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e812fd20 con 0x7fe7e8135300 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80a0e90 con 0x7fe7e80a1230 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e812e140 con 0x7fe7e806e9d0 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7ed342700 10 monclient(hunting): authenticate will time out at 1880.078443s 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7e6ffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe7e806e9d0 0x7fe7e80a0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:42692/0 (socket says 172.21.15.112:42692) 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.232+0000 7fe7e6ffd700 1 -- 172.21.15.112:0/28428898 learned_addr learned my addr 172.21.15.112:0/28428898 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7e77fe700 1 -- 172.21.15.112:0/28428898 >> v1:172.21.15.112:6789/0 conn(0x7fe7e8135300 legacy=0x7fe7e814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7e6ffd700 1 -- 172.21.15.112:0/28428898 >> v1:172.21.15.112:6790/0 conn(0x7fe7e806e9d0 legacy=0x7fe7e80a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 client.?.objecter ms_handle_connect 0x7fe7e8135300 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 client.?.objecter ms_handle_connect 0x7fe7e806e9d0 2024-09-18T12:01:48.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7de7fc700 1 -- 172.21.15.112:0/28428898 >> v1:172.21.15.129:6789/0 conn(0x7fe7e80a1230 legacy=0x7fe7e80a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 client.?.objecter ms_handle_connect 0x7fe7e80a1230 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484618213 0 0) 0x7fe7e812fd20 con 0x7fe7e8135300 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): my global_id is 134325 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c8004f10 con 0x7fe7e8135300 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748977669 0 0) 0x7fe7e812e140 con 0x7fe7e806e9d0 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): my global_id is 154124 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e812fd20 con 0x7fe7e806e9d0 2024-09-18T12:01:48.248 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950944065 0 0) 0x7fe7e80a0e90 con 0x7fe7e80a1230 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): my global_id is 144208 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e812e140 con 0x7fe7e80a1230 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2680876328 0 0) 0x7fe7c8004f10 con 0x7fe7e8135300 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7e80a0e90 con 0x7fe7e8135300 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7e0004d50 con 0x7fe7e8135300 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): dump: 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.249 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580046297 0 0) 0x7fe7e812fd20 con 0x7fe7e806e9d0 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7c8004f10 con 0x7fe7e806e9d0 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7d8001cb0 con 0x7fe7e806e9d0 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): dump: 2024-09-18T12:01:48.250 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396219590 0 0) 0x7fe7e812e140 con 0x7fe7e80a1230 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7e812fd20 con 0x7fe7e80a1230 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7d0001580 con 0x7fe7e80a1230 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.251 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient(hunting): dump: 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1785453263 0 0) 0x7fe7e80a0e90 con 0x7fe7e8135300 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 >> v1:172.21.15.112:6790/0 conn(0x7fe7e806e9d0 legacy=0x7fe7e80a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 >> v1:172.21.15.129:6789/0 conn(0x7fe7e80a1230 legacy=0x7fe7e80a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 monclient: found mon.a 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e812eac0 con 0x7fe7e8135300 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient: _finish_auth 0 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 5 monclient: authenticate success, global_id 134325 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 10 monclient: _renew_subs 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e812f9b0 con 0x7fe7e8135300 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 10 client.134325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 10 monclient: _renew_subs 2024-09-18T12:01:48.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e809b690 con 0x7fe7e8135300 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 1 librados: init done 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7e0005160 con 0x7fe7e8135300 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7e0001940 con 0x7fe7e8135300 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7e5ffb700 4 set_mon_vals no callback set 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7df7fe700 10 monclient: dump: 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:48.254 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.236+0000 7fe7ed342700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7ed342700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7ed342700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7cc0050b0 con 0x7fe7e8135300 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe7e000fb90 con 0x7fe7e8135300 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(689..689 src has 1..689) v4 ==== 5806+0+0 (unknown 2101935416 0 0) 0x7fe7e0044380 con 0x7fe7e8135300 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7df7fe700 10 client.134325.objecter ms_dispatch 0x7fe7e8000fd0 osd_map(689..689 src has 1..689) v4 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7df7fe700 3 client.134325.objecter handle_osd_map got epochs [689,689] > 0 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7df7fe700 3 client.134325.objecter handle_osd_map decoding full epoch 689 2024-09-18T12:01:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7df7fe700 20 client.134325.objecter dump_active .. 0 homeless 2024-09-18T12:01:48.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7e6ffd700 10 monclient: get_auth_request con 0x7fe7c8038560 auth_method 0 2024-09-18T12:01:48.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.240+0000 7fe7df7fe700 10 client.134325.objecter ms_handle_connect 0x7fe7c8038560 2024-09-18T12:01:48.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.248+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fe7e0014e10 con 0x7fe7e8135300 2024-09-18T12:01:48.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.248+0000 7fe7df7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:48.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.248+0000 7fe7df7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:48.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.472+0000 7fe7ed342700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:01:48.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.472+0000 7fe7ed342700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:01:48.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.472+0000 7fe7ed342700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:01:48.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.472+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe7cc005c30 con 0x7fe7e8135300 2024-09-18T12:01:48.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.472+0000 7fe7df7fe700 1 -- 172.21.15.112:0/28428898 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fe7e0014810 con 0x7fe7e8135300 2024-09-18T12:01:48.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.472+0000 7fe7df7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:01:48.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.472+0000 7fe7df7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:48.479 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.479 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=8,logm=57} crc {auth=1335469701,config=3140514009,health=1774101977,kv=1310296871,logm=810814749}) 2024-09-18T12:01:48.480 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=8,logm=55} crc {auth=2042959921,config=3140514009,health=1774101977,kv=1310296871,logm=1431197671}) 2024-09-18T12:01:48.480 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.480 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=8,logm=57} crc {auth=1335469701,config=3140514009,health=1774101977,kv=1310296871,logm=810814749}) 2024-09-18T12:01:48.480 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=9,kv=8,logm=58} crc {auth=187497960,config=3140514009,health=886635089,kv=1310296871,logm=219822840}) 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 10 librados: watch_flush enter 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 10 librados: watch_flush exit 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 20 client.134325.objecter shutdown clearing up homeless session... 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 10 client.134325.objecter successfully canceled tick 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 >> v1:172.21.15.129:6812/20806 conn(0x7fe7c8038560 legacy=0x7fe7c803aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 10 monclient: shutdown 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 >> v1:172.21.15.112:6789/0 conn(0x7fe7e8135300 legacy=0x7fe7e814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 shutdown_connections 2024-09-18T12:01:48.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 >> 172.21.15.112:0/28428898 conn(0x7fe7e80680d0 msgr2=0x7fe7e81345c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:48.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 shutdown_connections 2024-09-18T12:01:48.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 1 -- 172.21.15.112:0/28428898 wait complete. 2024-09-18T12:01:48.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fe7ed342700 1 librados: shutdown 2024-09-18T12:01:48.483 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.483 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1072412501}) 2024-09-18T12:01:48.483 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1356656289}) 2024-09-18T12:01:48.483 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.483 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1072412501}) 2024-09-18T12:01:48.483 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.476+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=345182799}) 2024-09-18T12:01:48.486 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.480+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.486 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.480+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554680779}) 2024-09-18T12:01:48.486 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.480+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638639711}) 2024-09-18T12:01:48.486 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.480+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.486 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.480+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554680779}) 2024-09-18T12:01:48.486 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.480+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1622629043}) 2024-09-18T12:01:48.489 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.484+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.489 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.484+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142737684}) 2024-09-18T12:01:48.489 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.484+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1561726498}) 2024-09-18T12:01:48.490 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.484+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.490 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.484+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142737684}) 2024-09-18T12:01:48.490 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.484+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4116390210}) 2024-09-18T12:01:48.491 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.491 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30111406}) 2024-09-18T12:01:48.491 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1974199623}) 2024-09-18T12:01:48.491 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.491 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30111406}) 2024-09-18T12:01:48.491 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536394474}) 2024-09-18T12:01:48.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:01:48.493 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.493 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156599504}) 2024-09-18T12:01:48.493 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2523005390}) 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156599504}) 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195178877}) 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741959585}) 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2535088007}) 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741959585}) 2024-09-18T12:01:48.494 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.488+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433851781}) 2024-09-18T12:01:48.495 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.495 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2782234516}) 2024-09-18T12:01:48.495 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=406814936}) 2024-09-18T12:01:48.495 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.495 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2782234516}) 2024-09-18T12:01:48.495 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1507079578}) 2024-09-18T12:01:48.496 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.496 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3980968153}) 2024-09-18T12:01:48.496 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2701411951}) 2024-09-18T12:01:48.496 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.496 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3980968153}) 2024-09-18T12:01:48.496 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=131131550}) 2024-09-18T12:01:48.497 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.497 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075541955}) 2024-09-18T12:01:48.497 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=172455459}) 2024-09-18T12:01:48.497 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.497 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075541955}) 2024-09-18T12:01:48.497 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174536050}) 2024-09-18T12:01:48.498 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.499 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437459138}) 2024-09-18T12:01:48.499 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727492926}) 2024-09-18T12:01:48.499 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.499 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437459138}) 2024-09-18T12:01:48.499 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.492+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=555514687}) 2024-09-18T12:01:48.500 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.500 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541492299}) 2024-09-18T12:01:48.500 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875870624}) 2024-09-18T12:01:48.500 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.500 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541492299}) 2024-09-18T12:01:48.500 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=694898999}) 2024-09-18T12:01:48.501 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.501 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049444606}) 2024-09-18T12:01:48.501 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2811902163}) 2024-09-18T12:01:48.501 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.501 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049444606}) 2024-09-18T12:01:48.501 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=77402025}) 2024-09-18T12:01:48.502 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.502 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103109134}) 2024-09-18T12:01:48.502 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=252153153}) 2024-09-18T12:01:48.503 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.503 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103109134}) 2024-09-18T12:01:48.503 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.496+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2437291470}) 2024-09-18T12:01:48.504 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.504 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704028589}) 2024-09-18T12:01:48.504 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583817076}) 2024-09-18T12:01:48.504 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.504 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704028589}) 2024-09-18T12:01:48.504 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3591352912}) 2024-09-18T12:01:48.506 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.506 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254181787}) 2024-09-18T12:01:48.506 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1134985756}) 2024-09-18T12:01:48.506 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.506 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254181787}) 2024-09-18T12:01:48.506 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.500+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2418999818}) 2024-09-18T12:01:48.507 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.507 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163540826}) 2024-09-18T12:01:48.507 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1711798315}) 2024-09-18T12:01:48.507 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.508 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163540826}) 2024-09-18T12:01:48.508 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536002967}) 2024-09-18T12:01:48.509 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.509 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436078250}) 2024-09-18T12:01:48.509 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2321354458}) 2024-09-18T12:01:48.509 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.509 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436078250}) 2024-09-18T12:01:48.510 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290033142}) 2024-09-18T12:01:48.511 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.511 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177701591}) 2024-09-18T12:01:48.511 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3946867359}) 2024-09-18T12:01:48.511 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.511 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177701591}) 2024-09-18T12:01:48.511 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.504+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845443511}) 2024-09-18T12:01:48.512 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.512 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237223145}) 2024-09-18T12:01:48.512 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62492220}) 2024-09-18T12:01:48.512 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.512 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237223145}) 2024-09-18T12:01:48.512 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056128671}) 2024-09-18T12:01:48.514 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.514 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2699093236}) 2024-09-18T12:01:48.514 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912227786}) 2024-09-18T12:01:48.514 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.514 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2699093236}) 2024-09-18T12:01:48.514 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701559920}) 2024-09-18T12:01:48.515 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.515 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508745831}) 2024-09-18T12:01:48.515 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.508+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3065917729}) 2024-09-18T12:01:48.515 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.515 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508745831}) 2024-09-18T12:01:48.515 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310348747}) 2024-09-18T12:01:48.516 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.516 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778727580}) 2024-09-18T12:01:48.516 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899255739}) 2024-09-18T12:01:48.516 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.516 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778727580}) 2024-09-18T12:01:48.517 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293016705}) 2024-09-18T12:01:48.517 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.518 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255620865}) 2024-09-18T12:01:48.518 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2134488409}) 2024-09-18T12:01:48.518 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.518 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255620865}) 2024-09-18T12:01:48.518 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.512+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1123192703}) 2024-09-18T12:01:48.519 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.519 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1845340084}) 2024-09-18T12:01:48.519 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1392611455}) 2024-09-18T12:01:48.519 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.519 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1845340084}) 2024-09-18T12:01:48.519 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126663687}) 2024-09-18T12:01:48.520 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.520 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270138252}) 2024-09-18T12:01:48.521 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519398745}) 2024-09-18T12:01:48.521 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.521 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270138252}) 2024-09-18T12:01:48.521 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078024499}) 2024-09-18T12:01:48.522 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.522 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867777036}) 2024-09-18T12:01:48.522 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3486456149}) 2024-09-18T12:01:48.522 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.522 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867777036}) 2024-09-18T12:01:48.522 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.516+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1861172906}) 2024-09-18T12:01:48.523 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.523 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229789564}) 2024-09-18T12:01:48.523 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879730368}) 2024-09-18T12:01:48.523 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.523 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229789564}) 2024-09-18T12:01:48.524 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246794830}) 2024-09-18T12:01:48.524 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406107234}) 2024-09-18T12:01:48.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422186824}) 2024-09-18T12:01:48.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406107234}) 2024-09-18T12:01:48.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062545871}) 2024-09-18T12:01:48.526 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.526 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356651544}) 2024-09-18T12:01:48.526 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2191946513}) 2024-09-18T12:01:48.526 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.526 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356651544}) 2024-09-18T12:01:48.526 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.520+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461467316}) 2024-09-18T12:01:48.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287123978}) 2024-09-18T12:01:48.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917912340}) 2024-09-18T12:01:48.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287123978}) 2024-09-18T12:01:48.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1105609519}) 2024-09-18T12:01:48.528 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.528 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503380688}) 2024-09-18T12:01:48.528 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2791659815}) 2024-09-18T12:01:48.528 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.528 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503380688}) 2024-09-18T12:01:48.528 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1454253299}) 2024-09-18T12:01:48.529 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.529 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399451174}) 2024-09-18T12:01:48.529 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1159710079}) 2024-09-18T12:01:48.529 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.529 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399451174}) 2024-09-18T12:01:48.529 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369388972}) 2024-09-18T12:01:48.530 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.530 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781779641}) 2024-09-18T12:01:48.530 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3759745010}) 2024-09-18T12:01:48.530 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.530 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781779641}) 2024-09-18T12:01:48.530 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.524+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1293410024}) 2024-09-18T12:01:48.531 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.531 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122561784}) 2024-09-18T12:01:48.531 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112697379}) 2024-09-18T12:01:48.532 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.532 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122561784}) 2024-09-18T12:01:48.532 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51759957}) 2024-09-18T12:01:48.532 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.533 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287060706}) 2024-09-18T12:01:48.533 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=290086840}) 2024-09-18T12:01:48.533 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.533 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287060706}) 2024-09-18T12:01:48.533 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3844433026}) 2024-09-18T12:01:48.534 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.534 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575033904}) 2024-09-18T12:01:48.534 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2953835198}) 2024-09-18T12:01:48.534 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.534 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575033904}) 2024-09-18T12:01:48.534 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.528+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4165193572}) 2024-09-18T12:01:48.535 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.535 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040270279}) 2024-09-18T12:01:48.535 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309394807}) 2024-09-18T12:01:48.535 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.535 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040270279}) 2024-09-18T12:01:48.535 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915152134}) 2024-09-18T12:01:48.536 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.536 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872420792}) 2024-09-18T12:01:48.536 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2060172908}) 2024-09-18T12:01:48.536 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.537 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872420792}) 2024-09-18T12:01:48.537 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1543349452}) 2024-09-18T12:01:48.537 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.537 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864217644}) 2024-09-18T12:01:48.538 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1966154864}) 2024-09-18T12:01:48.538 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.538 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864217644}) 2024-09-18T12:01:48.538 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2386457649}) 2024-09-18T12:01:48.539 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.539 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1958644918}) 2024-09-18T12:01:48.539 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933492366}) 2024-09-18T12:01:48.539 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.539 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1958644918}) 2024-09-18T12:01:48.539 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.532+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763944233}) 2024-09-18T12:01:48.540 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.540 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893093070}) 2024-09-18T12:01:48.540 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2099977401}) 2024-09-18T12:01:48.540 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.540 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893093070}) 2024-09-18T12:01:48.540 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1344452036}) 2024-09-18T12:01:48.541 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.541 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909953669}) 2024-09-18T12:01:48.541 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797799927}) 2024-09-18T12:01:48.542 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.542 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909953669}) 2024-09-18T12:01:48.542 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1596197703}) 2024-09-18T12:01:48.543 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.543 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383150520}) 2024-09-18T12:01:48.543 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350431852}) 2024-09-18T12:01:48.543 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.543 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383150520}) 2024-09-18T12:01:48.543 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.536+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2250955284}) 2024-09-18T12:01:48.544 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.544 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499384613}) 2024-09-18T12:01:48.544 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686433290}) 2024-09-18T12:01:48.544 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.544 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499384613}) 2024-09-18T12:01:48.544 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3390135999}) 2024-09-18T12:01:48.545 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.545 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341443659}) 2024-09-18T12:01:48.545 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107504942}) 2024-09-18T12:01:48.545 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.545 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341443659}) 2024-09-18T12:01:48.545 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613708421}) 2024-09-18T12:01:48.546 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.546 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406035385}) 2024-09-18T12:01:48.546 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3691556388}) 2024-09-18T12:01:48.546 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.546 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406035385}) 2024-09-18T12:01:48.546 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.540+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=227363406}) 2024-09-18T12:01:48.547 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.547 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307856144}) 2024-09-18T12:01:48.547 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2546594075}) 2024-09-18T12:01:48.547 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.547 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307856144}) 2024-09-18T12:01:48.547 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127492491}) 2024-09-18T12:01:48.548 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.548 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513819502}) 2024-09-18T12:01:48.548 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1758524579}) 2024-09-18T12:01:48.548 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.548 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513819502}) 2024-09-18T12:01:48.549 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=148446651}) 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596949782}) 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398578930}) 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596949782}) 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643717980}) 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.544+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561494025}) 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462047024}) 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.554 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561494025}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=703533237}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143959207}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683602410}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143959207}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3862805064}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337076855}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1098421423}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337076855}) 2024-09-18T12:01:48.555 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1703134259}) 2024-09-18T12:01:48.557 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.557 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70021978}) 2024-09-18T12:01:48.557 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4289786307}) 2024-09-18T12:01:48.557 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.548+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.557 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70021978}) 2024-09-18T12:01:48.557 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=649520053}) 2024-09-18T12:01:48.557 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544064663}) 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1269800981}) 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544064663}) 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235061686}) 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323502204}) 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1147683262}) 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323502204}) 2024-09-18T12:01:48.558 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929411326}) 2024-09-18T12:01:48.559 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.559 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3237973868}) 2024-09-18T12:01:48.559 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056055688}) 2024-09-18T12:01:48.559 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.560 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3237973868}) 2024-09-18T12:01:48.560 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.552+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3416399049}) 2024-09-18T12:01:48.560 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.560 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650241711}) 2024-09-18T12:01:48.560 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335232949}) 2024-09-18T12:01:48.561 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.561 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650241711}) 2024-09-18T12:01:48.561 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2248643477}) 2024-09-18T12:01:48.561 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.561 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455412299}) 2024-09-18T12:01:48.562 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3985709234}) 2024-09-18T12:01:48.562 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.562 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455412299}) 2024-09-18T12:01:48.562 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3636018349}) 2024-09-18T12:01:48.562 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.563 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623714325}) 2024-09-18T12:01:48.563 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061936720}) 2024-09-18T12:01:48.563 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.563 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623714325}) 2024-09-18T12:01:48.563 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.556+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763336263}) 2024-09-18T12:01:48.564 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.564 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859941814}) 2024-09-18T12:01:48.564 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=643396160}) 2024-09-18T12:01:48.564 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.564 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859941814}) 2024-09-18T12:01:48.564 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2231041085}) 2024-09-18T12:01:48.565 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.565 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8721879}) 2024-09-18T12:01:48.565 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4242287234}) 2024-09-18T12:01:48.565 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.565 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8721879}) 2024-09-18T12:01:48.565 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967685077}) 2024-09-18T12:01:48.566 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.566 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123144941}) 2024-09-18T12:01:48.566 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4294021942}) 2024-09-18T12:01:48.566 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.567 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123144941}) 2024-09-18T12:01:48.567 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.560+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=217963429}) 2024-09-18T12:01:48.568 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.568 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090965873}) 2024-09-18T12:01:48.568 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=118993499}) 2024-09-18T12:01:48.568 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.568 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090965873}) 2024-09-18T12:01:48.568 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3278129164}) 2024-09-18T12:01:48.569 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.569 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275411582}) 2024-09-18T12:01:48.569 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1232875031}) 2024-09-18T12:01:48.569 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.569 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275411582}) 2024-09-18T12:01:48.569 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1037161885}) 2024-09-18T12:01:48.570 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.570 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635763199}) 2024-09-18T12:01:48.570 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=456115430}) 2024-09-18T12:01:48.571 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.571 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635763199}) 2024-09-18T12:01:48.571 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.564+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2777139048}) 2024-09-18T12:01:48.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.672+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.672+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=1} crc {logm=2130128106,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=3958603105,mon_config_key=868353578,monmap=1228700967}) 2024-09-18T12:01:48.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.672+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=5} crc {logm=2072687438,mdsmap=3140514009,mgr=4129098723,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=209861722,mon_config_key=868353578,monmap=1725997317,osd_metadata=1236761652}) 2024-09-18T12:01:48.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.672+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.679 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.672+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=1} crc {logm=2130128106,mdsmap=1531151339,mgr=3797927047,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=3958603105,mon_config_key=868353578,monmap=1228700967}) 2024-09-18T12:01:48.679 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.672+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=2,osd_metadata=4} crc {logm=2662593415,mdsmap=1531151339,mgr=3772577836,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=1681124417,mon_config_key=868353578,monmap=3140514009,osd_metadata=3422354632}) 2024-09-18T12:01:48.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.684+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.684+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=44,osdmap=48} crc {monmap=1228700967,osd_metadata=871764258,osd_pg_creating=365523528,osd_snap=469830625,osdmap=1981646795}) 2024-09-18T12:01:48.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.684+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=45,osdmap=47} crc {monmap=1228700967,osd_metadata=871764258,osd_pg_creating=365523528,osd_snap=525655151,osdmap=1146825014}) 2024-09-18T12:01:48.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.684+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.684+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=44,osdmap=48} crc {monmap=1228700967,osd_metadata=871764258,osd_pg_creating=365523528,osd_snap=469830625,osdmap=1981646795}) 2024-09-18T12:01:48.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.684+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=1,osd_metadata=5,osd_pg_creating=1,osd_snap=43,osdmap=50} crc {monmap=1228700967,osd_metadata=3426400433,osd_pg_creating=365523528,osd_snap=105109758,osdmap=2033571361}) 2024-09-18T12:01:48.745 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.740+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.746 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.740+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=430189674}) 2024-09-18T12:01:48.746 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.740+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2956668329}) 2024-09-18T12:01:48.746 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.740+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.746 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.740+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=430189674}) 2024-09-18T12:01:48.746 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.740+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2279392678}) 2024-09-18T12:01:48.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.744+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.744+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=29348571}) 2024-09-18T12:01:48.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.744+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1193187142}) 2024-09-18T12:01:48.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.744+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:48.749 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.744+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=29348571}) 2024-09-18T12:01:48.750 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:48.744+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2722416083}) 2024-09-18T12:01:49.259 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.259 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=866966851}) 2024-09-18T12:01:49.259 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3301395392}) 2024-09-18T12:01:49.260 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.260 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=866966851}) 2024-09-18T12:01:49.260 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3792637450}) 2024-09-18T12:01:49.262 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.262 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4255555228}) 2024-09-18T12:01:49.262 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2760852338}) 2024-09-18T12:01:49.262 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.262 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4255555228}) 2024-09-18T12:01:49.262 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.256+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1659549030}) 2024-09-18T12:01:49.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.260+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.260+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1161659769}) 2024-09-18T12:01:49.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.260+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4160875588}) 2024-09-18T12:01:49.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.260+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.260+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1161659769}) 2024-09-18T12:01:49.265 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.260+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3016024708}) 2024-09-18T12:01:49.267 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.264+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.264+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=478934911}) 2024-09-18T12:01:49.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.264+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=13058887}) 2024-09-18T12:01:49.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.264+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.264+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=478934911}) 2024-09-18T12:01:49.268 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.264+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2597900337}) 2024-09-18T12:01:49.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.268+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.268+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=884125164}) 2024-09-18T12:01:49.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.268+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1434449057}) 2024-09-18T12:01:49.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.268+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.268+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=884125164}) 2024-09-18T12:01:49.272 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.268+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2785123260}) 2024-09-18T12:01:49.276 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.276 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3975544297}) 2024-09-18T12:01:49.276 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3090747487}) 2024-09-18T12:01:49.276 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.276 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3975544297}) 2024-09-18T12:01:49.276 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2564884581}) 2024-09-18T12:01:49.278 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.279 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3931260110}) 2024-09-18T12:01:49.279 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1408589901}) 2024-09-18T12:01:49.279 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.279 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3931260110}) 2024-09-18T12:01:49.279 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.272+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2046701549}) 2024-09-18T12:01:49.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.280+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.280+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=2665253032}) 2024-09-18T12:01:49.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.280+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=63} crc {osdmap=4260779840}) 2024-09-18T12:01:49.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.280+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:01:49.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.280+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=2665253032}) 2024-09-18T12:01:49.285 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:01:49.280+0000 7fd9ca5ef700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=513592513}) 2024-09-18T12:01:49.494 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:01:49.494 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:01:49.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient: get_monmap_and_config 2024-09-18T12:01:49.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient: build_initial_monmap 2024-09-18T12:01:49.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient: monmap: 2024-09-18T12:01:49.596 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:49.596 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.596 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:49.596990+0000 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:49.596990+0000 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 1 Processor -- start 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 1 -- start start 2024-09-18T12:01:49.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient: init 2024-09-18T12:01:49.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:49.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:01:49.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb8b40a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:49.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb8b406c280 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:49.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb8b406c670 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:49.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b406e950 con 0x7fb8b40a1ea0 2024-09-18T12:01:49.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b40a2270 con 0x7fb8b406c670 2024-09-18T12:01:49.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b406d130 con 0x7fb8b406c280 2024-09-18T12:01:49.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:49.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b3fff700 10 monclient(hunting): authenticate will time out at 1881.437812s 2024-09-18T12:01:49.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b2ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb8b40a1ea0 0x7fb8b406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56448/0 (socket says 172.21.15.112:56448) 2024-09-18T12:01:49.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8b2ffd700 1 -- 172.21.15.112:0/2316538866 learned_addr learned my addr 172.21.15.112:0/2316538866 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:49.599 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.592+0000 7fb8aa7fc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb8b406c280 0x7fb8b4135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52858/0 (socket says 172.21.15.112:52858) 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b2ffd700 1 -- 172.21.15.112:0/2316538866 >> v1:172.21.15.112:6789/0 conn(0x7fb8b40a1ea0 legacy=0x7fb8b406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8aa7fc700 1 -- 172.21.15.112:0/2316538866 >> v1:172.21.15.112:6790/0 conn(0x7fb8b406c280 legacy=0x7fb8b4135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249579575 0 0) 0x7fb8b406e950 con 0x7fb8b40a1ea0 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient(hunting): my global_id is 134328 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a0003570 con 0x7fb8b40a1ea0 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623031117 0 0) 0x7fb8b406d130 con 0x7fb8b406c280 2024-09-18T12:01:49.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient(hunting): my global_id is 154127 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b406e950 con 0x7fb8b406c280 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b37fe700 1 -- 172.21.15.112:0/2316538866 >> v1:172.21.15.129:6789/0 conn(0x7fb8b406c670 legacy=0x7fb8b41388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973908494 0 0) 0x7fb8b406e950 con 0x7fb8b406c280 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 >> v1:172.21.15.129:6789/0 conn(0x7fb8b406c670 legacy=0x7fb8b41388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 >> v1:172.21.15.112:6789/0 conn(0x7fb8b40a1ea0 legacy=0x7fb8b406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: _finish_hunting 0 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 monclient: found mon.noname-b 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b409e370 con 0x7fb8b406c280 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: _finish_auth 0 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 5 monclient: authenticate success, global_id 154127 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb89c001920 con 0x7fb8b406c280 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: dump: 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.604 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb89c001c70 con 0x7fb8b406c280 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:49.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 1 -- 172.21.15.112:0/2316538866 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb89c0049d0 con 0x7fb8b406c280 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 4 set_mon_vals no callback set 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b27fc700 10 monclient: dump: 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.606 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient: shutdown 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 -- 172.21.15.112:0/2316538866 >> v1:172.21.15.112:6790/0 conn(0x7fb8b406c280 legacy=0x7fb8b4135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 -- 172.21.15.112:0/2316538866 shutdown_connections 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 -- 172.21.15.112:0/2316538866 >> 172.21.15.112:0/2316538866 conn(0x7fb8b4067da0 msgr2=0x7fb8b40a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 -- 172.21.15.112:0/2316538866 shutdown_connections 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 -- 172.21.15.112:0/2316538866 wait complete. 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient: build_initial_monmap 2024-09-18T12:01:49.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient: monmap: 2024-09-18T12:01:49.608 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:49.608 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:49.608 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:49.608 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:49.608 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:49.608 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.609 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:49.609 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:49.609 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:49.609 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 librados: starting msgr at 2024-09-18T12:01:49.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 librados: starting objecter 2024-09-18T12:01:49.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 Processor -- start 2024-09-18T12:01:49.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 -- start start 2024-09-18T12:01:49.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 librados: setting wanted keys 2024-09-18T12:01:49.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 1 librados: calling monclient init 2024-09-18T12:01:49.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient: init 2024-09-18T12:01:49.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:49.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:01:49.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb8b406c670 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb8b40a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.596+0000 7fb8b3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb8b413f850 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b411def0 con 0x7fb8b406c670 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b41203b0 con 0x7fb8b40a1ea0 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b41200d0 con 0x7fb8b413f850 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 10 monclient(hunting): authenticate will time out at 1881.444079s 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b2ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb8b406c670 0x7fb8b4131fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56454/0 (socket says 172.21.15.112:56454) 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b2ffd700 1 -- 172.21.15.112:0/4100809781 learned_addr learned my addr 172.21.15.112:0/4100809781 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b2ffd700 1 -- 172.21.15.112:0/4100809781 >> v1:172.21.15.112:6789/0 conn(0x7fb8b406c670 legacy=0x7fb8b4131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 client.?.objecter ms_handle_connect 0x7fb8b406c670 2024-09-18T12:01:49.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b37fe700 1 -- 172.21.15.112:0/4100809781 >> v1:172.21.15.112:6790/0 conn(0x7fb8b413f850 legacy=0x7fb8b41474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 client.?.objecter ms_handle_connect 0x7fb8b413f850 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 client.?.objecter ms_handle_connect 0x7fb8b40a1ea0 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8aa7fc700 1 -- 172.21.15.112:0/4100809781 >> v1:172.21.15.129:6789/0 conn(0x7fb8b40a1ea0 legacy=0x7fb8b413f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121076767 0 0) 0x7fb8b41200d0 con 0x7fb8b413f850 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): my global_id is 154130 2024-09-18T12:01:49.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894004f10 con 0x7fb8b413f850 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156388438 0 0) 0x7fb8b41203b0 con 0x7fb8b40a1ea0 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): my global_id is 144214 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b41200d0 con 0x7fb8b40a1ea0 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3218826973 0 0) 0x7fb894004f10 con 0x7fb8b413f850 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b41203b0 con 0x7fb8b413f850 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8ac001f00 con 0x7fb8b413f850 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:49.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): dump: 2024-09-18T12:01:49.618 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.618 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.618 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.618 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.618 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.618 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1469969640 0 0) 0x7fb8b41203b0 con 0x7fb8b413f850 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 >> v1:172.21.15.129:6789/0 conn(0x7fb8b40a1ea0 legacy=0x7fb8b413f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 >> v1:172.21.15.112:6789/0 conn(0x7fb8b406c670 legacy=0x7fb8b4131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 monclient: found mon.c 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:49.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4120690 con 0x7fb8b413f850 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient: _finish_auth 0 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 5 monclient: authenticate success, global_id 154130 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 10 monclient: _renew_subs 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b41326b0 con 0x7fb8b413f850 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 10 client.154130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 10 monclient: _renew_subs 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b411f380 con 0x7fb8b413f850 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b3fff700 1 librados: init done 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8ac002530 con 0x7fb8b413f850 2024-09-18T12:01:49.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb8ac0028c0 con 0x7fb8b413f850 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8ab7fe700 10 monclient: dump: 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.621 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.600+0000 7fb8b1ffb700 4 set_mon_vals no callback set 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb8ac002a20 con 0x7fb8b413f850 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8b3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8b3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8b3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8980050b0 con 0x7fb8b413f850 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(691..691 src has 1..691) v4 ==== 5793+0+0 (unknown 1608782316 0 0) 0x7fb8ac044230 con 0x7fb8b413f850 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8ab7fe700 10 client.154130.objecter ms_dispatch 0x7fb8b4000fd0 osd_map(691..691 src has 1..691) v4 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8ab7fe700 3 client.154130.objecter handle_osd_map got epochs [691,691] > 0 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8ab7fe700 3 client.154130.objecter handle_osd_map decoding full epoch 691 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8ab7fe700 20 client.154130.objecter dump_active .. 0 homeless 2024-09-18T12:01:49.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.604+0000 7fb8b2ffd700 10 monclient: get_auth_request con 0x7fb894038350 auth_method 0 2024-09-18T12:01:49.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.608+0000 7fb8ab7fe700 10 client.154130.objecter ms_handle_connect 0x7fb894038350 2024-09-18T12:01:49.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.612+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fb8ac002cd0 con 0x7fb8b413f850 2024-09-18T12:01:49.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.612+0000 7fb8ab7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:49.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.612+0000 7fb8ab7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:49.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.836+0000 7fb8b3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:01:49.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.836+0000 7fb8b3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:01:49.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.836+0000 7fb8b3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.836+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb8980058e0 con 0x7fb8b413f850 2024-09-18T12:01:49.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.836+0000 7fb8ab7fe700 1 -- 172.21.15.112:0/4100809781 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 547915068) 0x7fb8ac014690 con 0x7fb8b413f850 2024-09-18T12:01:49.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.836+0000 7fb8ab7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:01:49.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.836+0000 7fb8ab7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:49.841 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:01:49.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 10 librados: watch_flush enter 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 10 librados: watch_flush exit 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 20 client.154130.objecter shutdown clearing up homeless session... 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 10 client.154130.objecter successfully canceled tick 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 >> v1:172.21.15.129:6812/20806 conn(0x7fb894038350 legacy=0x7fb89403a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 10 monclient: shutdown 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 >> v1:172.21.15.112:6790/0 conn(0x7fb8b413f850 legacy=0x7fb8b41474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 shutdown_connections 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 >> 172.21.15.112:0/4100809781 conn(0x7fb8b4067da0 msgr2=0x7fb8b409bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 shutdown_connections 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 1 -- 172.21.15.112:0/4100809781 wait complete. 2024-09-18T12:01:49.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.840+0000 7fb8b3fff700 1 librados: shutdown 2024-09-18T12:01:49.854 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:01:49.854 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:01:49.854 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:01:49.854 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:01:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient: get_monmap_and_config 2024-09-18T12:01:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient: build_initial_monmap 2024-09-18T12:01:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient: monmap: 2024-09-18T12:01:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:49.956 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:49.957383+0000 2024-09-18T12:01:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:49.957383+0000 2024-09-18T12:01:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:49.957 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:49.957 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 1 Processor -- start 2024-09-18T12:01:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 1 -- start start 2024-09-18T12:01:49.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient: init 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient(hunting): picked mon.noname-c con 0x7fe3600a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient(hunting): picked mon.noname-b con 0x7fe36006c280 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient(hunting): picked mon.noname-a con 0x7fe36006c670 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36006e950 con 0x7fe36006c670 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3600a2270 con 0x7fe3600a1ea0 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36006d130 con 0x7fe36006c280 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:49.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36687b700 10 monclient(hunting): authenticate will time out at 1881.798334s 2024-09-18T12:01:49.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36607a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe36006c670 0x7fe3601388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56460/0 (socket says 172.21.15.112:56460) 2024-09-18T12:01:49.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.952+0000 7fe36607a700 1 -- 172.21.15.112:0/559483566 learned_addr learned my addr 172.21.15.112:0/559483566 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:49.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe365078700 1 -- 172.21.15.112:0/559483566 >> v1:172.21.15.112:6790/0 conn(0x7fe36006c280 legacy=0x7fe360135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36607a700 1 -- 172.21.15.112:0/559483566 >> v1:172.21.15.112:6789/0 conn(0x7fe36006c670 legacy=0x7fe3601388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229947643 0 0) 0x7fe36006d130 con 0x7fe36006c280 2024-09-18T12:01:49.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient(hunting): my global_id is 154133 2024-09-18T12:01:49.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe34c003570 con 0x7fe36006c280 2024-09-18T12:01:49.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe365879700 1 -- 172.21.15.112:0/559483566 >> v1:172.21.15.129:6789/0 conn(0x7fe3600a1ea0 legacy=0x7fe36006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443401158 0 0) 0x7fe36006e950 con 0x7fe36006c670 2024-09-18T12:01:49.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient(hunting): my global_id is 134331 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36006d130 con 0x7fe36006c670 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2486919733 0 0) 0x7fe34c003570 con 0x7fe36006c280 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 >> v1:172.21.15.129:6789/0 conn(0x7fe3600a1ea0 legacy=0x7fe36006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 >> v1:172.21.15.112:6789/0 conn(0x7fe36006c670 legacy=0x7fe3601388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: _finish_hunting 0 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 monclient: found mon.noname-b 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36009e370 con 0x7fe36006c280 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: _finish_auth 0 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:49.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 5 monclient: authenticate success, global_id 154133 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe35c001990 con 0x7fe36006c280 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: dump: 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe35c001c20 con 0x7fe36006c280 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 4 set_mon_vals no callback set 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 1 -- 172.21.15.112:0/559483566 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe35c0020a0 con 0x7fe36006c280 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe364877700 10 monclient: dump: 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.975 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient: shutdown 2024-09-18T12:01:49.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 -- 172.21.15.112:0/559483566 >> v1:172.21.15.112:6790/0 conn(0x7fe36006c280 legacy=0x7fe360135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 -- 172.21.15.112:0/559483566 shutdown_connections 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 -- 172.21.15.112:0/559483566 >> 172.21.15.112:0/559483566 conn(0x7fe360067da0 msgr2=0x7fe3600a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 -- 172.21.15.112:0/559483566 shutdown_connections 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 -- 172.21.15.112:0/559483566 wait complete. 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient: build_initial_monmap 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient: monmap: 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.977 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 librados: starting msgr at 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 librados: starting objecter 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 Processor -- start 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 -- start start 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 librados: setting wanted keys 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 librados: calling monclient init 2024-09-18T12:01:49.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient: init 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient(hunting): picked mon.noname-a con 0x7fe36006c280 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient(hunting): picked mon.noname-b con 0x7fe36006c670 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 10 monclient(hunting): picked mon.noname-c con 0x7fe3600a1ea0 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3601269f0 con 0x7fe36006c280 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.956+0000 7fe36687b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe360126710 con 0x7fe36006c670 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe360126430 con 0x7fe3600a1ea0 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 10 monclient(hunting): authenticate will time out at 1881.804018s 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36607a700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe3600a1ea0 0x7fe36011b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52888/0 (socket says 172.21.15.112:52888) 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36607a700 1 -- 172.21.15.112:0/1276041058 learned_addr learned my addr 172.21.15.112:0/1276041058 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:49.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36607a700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6790/0 conn(0x7fe3600a1ea0 legacy=0x7fe36011b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 client.?.objecter ms_handle_connect 0x7fe3600a1ea0 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 client.?.objecter ms_handle_connect 0x7fe36006c280 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe365879700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6789/0 conn(0x7fe36006c280 legacy=0x7fe36011a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831695307 0 0) 0x7fe360126430 con 0x7fe3600a1ea0 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): my global_id is 154136 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344004f10 con 0x7fe3600a1ea0 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 client.?.objecter ms_handle_connect 0x7fe36006c670 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:49.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe365078700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.129:6789/0 conn(0x7fe36006c670 legacy=0x7fe36011a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697970969 0 0) 0x7fe3601269f0 con 0x7fe36006c280 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): my global_id is 134334 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe360126430 con 0x7fe36006c280 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 810652567 0 0) 0x7fe344004f10 con 0x7fe3600a1ea0 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3601269f0 con 0x7fe3600a1ea0 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102175661 0 0) 0x7fe360126710 con 0x7fe36006c670 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): my global_id is 144220 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344004f10 con 0x7fe36006c670 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe358002070 con 0x7fe3600a1ea0 2024-09-18T12:01:49.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:49.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): dump: 2024-09-18T12:01:49.982 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.982 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.982 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.982 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.984 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.984 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.984 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.984 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222786862 0 0) 0x7fe360126430 con 0x7fe36006c280 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe360126710 con 0x7fe36006c280 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe350001b80 con 0x7fe36006c280 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient(hunting): dump: 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.985 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1735932549 0 0) 0x7fe3601269f0 con 0x7fe3600a1ea0 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.129:6789/0 conn(0x7fe36006c670 legacy=0x7fe36011a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6789/0 conn(0x7fe36006c280 legacy=0x7fe36011a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 monclient: found mon.c 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3601254d0 con 0x7fe3600a1ea0 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 10 monclient: _finish_auth 0 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe355ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 5 monclient: authenticate success, global_id 154136 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 10 monclient: _renew_subs 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3601270f0 con 0x7fe3600a1ea0 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 10 client.154136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 10 monclient: _renew_subs 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36014f130 con 0x7fe3600a1ea0 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.960+0000 7fe36687b700 1 librados: init done 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe358004ed0 con 0x7fe3600a1ea0 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:49.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe358005220 con 0x7fe3600a1ea0 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe357fff700 4 set_mon_vals no callback set 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 monclient: dump: 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.988 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe36687b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe358005380 con 0x7fe3600a1ea0 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe340006590 con 0x7fe340002840 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(691..691 src has 1..691) v4 ==== 5793+0+0 (unknown 1608782316 0 0) 0x7fe3580442f0 con 0x7fe3600a1ea0 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 client.154136.objecter ms_dispatch 0x7fe360000fd0 osd_map(691..691 src has 1..691) v4 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 3 client.154136.objecter handle_osd_map got epochs [691,691] > 0 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 3 client.154136.objecter handle_osd_map decoding full epoch 691 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 20 client.154136.objecter dump_active .. 0 homeless 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe365078700 10 monclient: get_auth_request con 0x7fe344038560 auth_method 0 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 client.154136.objecter ms_handle_connect 0x7fe340002840 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe365879700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6789/0 conn(0x7fe340002840 legacy=0x7fe340004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 client.154136.objecter resend_mon_ops 2024-09-18T12:01:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546301987 0 0) 0x7fe340006590 con 0x7fe340002840 2024-09-18T12:01:49.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 10 monclient(hunting): my global_id is 134337 2024-09-18T12:01:49.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3601269f0 con 0x7fe340002840 2024-09-18T12:01:49.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.964+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2774970234 0 0) 0x7fe3601269f0 con 0x7fe340002840 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe340006590 con 0x7fe340002840 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe350002680 con 0x7fe340002840 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 10 monclient: dump: 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.991 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 10 client.154136.objecter ms_handle_connect 0x7fe344038560 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1695116241 0 0) 0x7fe340006590 con 0x7fe340002840 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe34000a140 con 0x7fe340002840 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe3500027e0 con 0x7fe340002840 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6789/0 conn(0x7fe340002840 legacy=0x7fe340004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.992 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.968+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6789/0 conn(0x7fe340002840 legacy=0x7fe340004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe36687b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe340009650 con 0x7fe3400090c0 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe355ffb700 10 client.154136.objecter ms_handle_connect 0x7fe3400090c0 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe365879700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6789/0 conn(0x7fe3400090c0 legacy=0x7fe340008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe355ffb700 10 client.154136.objecter resend_mon_ops 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239162759 0 0) 0x7fe340009650 con 0x7fe3400090c0 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe355ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe355ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.980+0000 7fe355ffb700 10 monclient(hunting): my global_id is 134340 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe340006590 con 0x7fe3400090c0 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 154979653 0 0) 0x7fe340006590 con 0x7fe3400090c0 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe340009650 con 0x7fe3400090c0 2024-09-18T12:01:49.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe350005810 con 0x7fe3400090c0 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 10 monclient: dump: 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:49.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1882279477 0 0) 0x7fe340009650 con 0x7fe3400090c0 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe340009910 con 0x7fe3400090c0 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4173 (unknown 0 0 516864150) 0x7fe350005970 con 0x7fe3400090c0 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6789/0 conn(0x7fe3400090c0 legacy=0x7fe340008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe355ffb700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6789/0 conn(0x7fe3400090c0 legacy=0x7fe340008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T12:01:49.996 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 76, 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:49.997 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:49.998 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:49.999 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:50.000 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:01:50.003 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.004 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.005 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.006 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:01:50.007 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:01:50.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 10 librados: watch_flush enter 2024-09-18T12:01:50.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 10 librados: watch_flush exit 2024-09-18T12:01:50.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 20 client.154136.objecter shutdown clearing up homeless session... 2024-09-18T12:01:50.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 10 client.154136.objecter successfully canceled tick 2024-09-18T12:01:50.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.129:6812/20806 conn(0x7fe344038560 legacy=0x7fe34403aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 10 monclient: shutdown 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36607a700 1 -- 172.21.15.112:0/1276041058 reap_dead start 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 >> v1:172.21.15.112:6790/0 conn(0x7fe3600a1ea0 legacy=0x7fe36011b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 shutdown_connections 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 >> 172.21.15.112:0/1276041058 conn(0x7fe360067da0 msgr2=0x7fe360099f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 shutdown_connections 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 1 -- 172.21.15.112:0/1276041058 wait complete. 2024-09-18T12:01:50.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:49.984+0000 7fe36687b700 1 librados: shutdown 2024-09-18T12:01:50.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:01:50.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:01:50.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:01:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient: get_monmap_and_config 2024-09-18T12:01:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient: build_initial_monmap 2024-09-18T12:01:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient: monmap: 2024-09-18T12:01:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:50.124460+0000 2024-09-18T12:01:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:50.124460+0000 2024-09-18T12:01:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:50.124 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 1 Processor -- start 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 1 -- start start 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient: init 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f1d0c09ab20 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f1d0c099010 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f1d0c099410 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c0a01d0 con 0x7f1d0c099010 2024-09-18T12:01:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c09a440 con 0x7f1d0c099410 2024-09-18T12:01:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c098950 con 0x7f1d0c09ab20 2024-09-18T12:01:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0bfff700 10 monclient(hunting): authenticate will time out at 1881.965227s 2024-09-18T12:01:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0affd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1d0c09ab20 0x7f1d0c09aef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52894/0 (socket says 172.21.15.112:52894) 2024-09-18T12:01:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0affd700 1 -- 172.21.15.112:0/2437732179 learned_addr learned my addr 172.21.15.112:0/2437732179 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:50.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0affd700 1 -- 172.21.15.112:0/2437732179 >> v1:172.21.15.112:6790/0 conn(0x7f1d0c09ab20 legacy=0x7f1d0c09aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0a7fc700 1 -- 172.21.15.112:0/2437732179 >> v1:172.21.15.112:6789/0 conn(0x7f1d0c099010 legacy=0x7f1d0c13b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.120+0000 7f1d0b7fe700 1 -- 172.21.15.112:0/2437732179 >> v1:172.21.15.129:6789/0 conn(0x7f1d0c099410 legacy=0x7f1d0c13ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984948346 0 0) 0x7f1d0c0a01d0 con 0x7f1d0c099010 2024-09-18T12:01:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): my global_id is 134343 2024-09-18T12:01:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cec003570 con 0x7f1d0c099010 2024-09-18T12:01:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002004417 0 0) 0x7f1d0c098950 con 0x7f1d0c09ab20 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): my global_id is 154139 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d0c0a01d0 con 0x7f1d0c09ab20 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578687144 0 0) 0x7f1d0c09a440 con 0x7f1d0c099410 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient(hunting): my global_id is 144223 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d0c098950 con 0x7f1d0c099410 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2879492491 0 0) 0x7f1cec003570 con 0x7f1d0c099010 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 >> v1:172.21.15.112:6790/0 conn(0x7f1d0c09ab20 legacy=0x7f1d0c09aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 >> v1:172.21.15.129:6789/0 conn(0x7f1d0c099410 legacy=0x7f1d0c13ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:01:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 monclient: found mon.noname-a 2024-09-18T12:01:50.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:50.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:01:50.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c099b30 con 0x7f1d0c099010 2024-09-18T12:01:50.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: _finish_auth 0 2024-09-18T12:01:50.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:50.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1cf4001dd0 con 0x7f1d0c099010 2024-09-18T12:01:50.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 5 monclient: authenticate success, global_id 134343 2024-09-18T12:01:50.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:50.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:50.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: dump: 2024-09-18T12:01:50.134 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.134 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.134 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.134 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.135 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.135 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.135 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.135 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.135 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.135 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cf4001fb0 con 0x7f1d0c099010 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 1 -- 172.21.15.112:0/2437732179 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1cf4002420 con 0x7f1d0c099010 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d09ffb700 10 monclient: dump: 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 4 set_mon_vals no callback set 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient: shutdown 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2437732179 >> v1:172.21.15.112:6789/0 conn(0x7f1d0c099010 legacy=0x7f1d0c13b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2437732179 shutdown_connections 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2437732179 >> 172.21.15.112:0/2437732179 conn(0x7f1d0c0a1f90 msgr2=0x7f1d0c09f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2437732179 shutdown_connections 2024-09-18T12:01:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2437732179 wait complete. 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient: build_initial_monmap 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient: monmap: 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:50.144 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 librados: starting msgr at 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 librados: starting objecter 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 Processor -- start 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- start start 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 librados: setting wanted keys 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 librados: calling monclient init 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient: init 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f1d0c099010 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f1d0c099410 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:50.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f1d0c09ab20 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:50.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c13b230 con 0x7f1d0c09ab20 2024-09-18T12:01:50.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c1461c0 con 0x7f1d0c099010 2024-09-18T12:01:50.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c09b610 con 0x7f1d0c099410 2024-09-18T12:01:50.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:50.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.124+0000 7f1d0bfff700 10 monclient(hunting): authenticate will time out at 1881.971950s 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0a7fc700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f1d0c099410 0x7f1d0c13fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52902/0 (socket says 172.21.15.112:52902) 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0a7fc700 1 -- 172.21.15.112:0/2730910827 learned_addr learned my addr 172.21.15.112:0/2730910827 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0a7fc700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6790/0 conn(0x7f1d0c099410 legacy=0x7f1d0c13fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0b7fe700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6789/0 conn(0x7f1d0c09ab20 legacy=0x7f1d0c14de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 client.?.objecter ms_handle_connect 0x7f1d0c099410 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 client.?.objecter ms_handle_connect 0x7f1d0c09ab20 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 client.?.objecter ms_handle_connect 0x7f1d0c099010 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0affd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.129:6789/0 conn(0x7f1d0c099010 legacy=0x7f1d0c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627396571 0 0) 0x7f1d0c09b610 con 0x7f1d0c099410 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): my global_id is 154142 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf0004f10 con 0x7f1d0c099410 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877587178 0 0) 0x7f1d0c13b230 con 0x7f1d0c09ab20 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): my global_id is 134346 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d0c09b610 con 0x7f1d0c09ab20 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807703235 0 0) 0x7f1d0c1461c0 con 0x7f1d0c099010 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): my global_id is 144226 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d0c13b230 con 0x7f1d0c099010 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693546182 0 0) 0x7f1cf0004f10 con 0x7f1d0c099410 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d0c1461c0 con 0x7f1d0c099410 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1cf4001d10 con 0x7f1d0c099410 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): dump: 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.149 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2375095323 0 0) 0x7f1d0c09b610 con 0x7f1d0c09ab20 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cf0004f10 con 0x7f1d0c09ab20 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1d04001c00 con 0x7f1d0c09ab20 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:50.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient(hunting): dump: 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 515095111 0 0) 0x7f1d0c1461c0 con 0x7f1d0c099410 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.129:6789/0 conn(0x7f1d0c099010 legacy=0x7f1d0c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6789/0 conn(0x7f1d0c09ab20 legacy=0x7f1d0c14de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 monclient: found mon.c 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c13af50 con 0x7f1d0c099410 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient: _finish_auth 0 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 5 monclient: authenticate success, global_id 154142 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 10 monclient: _renew_subs 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c06eaf0 con 0x7f1d0c099410 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 10 client.154142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:50.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 10 monclient: _renew_subs 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c06df20 con 0x7f1d0c099410 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d0bfff700 1 librados: init done 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cf4002a80 con 0x7f1d0c099410 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1cf4002d90 con 0x7f1d0c099410 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d097fa700 4 set_mon_vals no callback set 2024-09-18T12:01:50.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.128+0000 7f1d02ffd700 10 monclient: dump: 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f1cf4002fa0 con 0x7f1d0c099410 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d0bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:50.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1ce4006590 con 0x7f1ce4002840 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(691..691 src has 1..691) v4 ==== 5793+0+0 (unknown 1608782316 0 0) 0x7f1cf4044590 con 0x7f1d0c099410 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d02ffd700 10 client.154142.objecter ms_dispatch 0x7f1d0c000fd0 osd_map(691..691 src has 1..691) v4 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d02ffd700 3 client.154142.objecter handle_osd_map got epochs [691,691] > 0 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d02ffd700 3 client.154142.objecter handle_osd_map decoding full epoch 691 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d02ffd700 20 client.154142.objecter dump_active .. 0 homeless 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.132+0000 7f1d0b7fe700 10 monclient: get_auth_request con 0x7f1cf0038270 auth_method 0 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 client.154142.objecter ms_handle_connect 0x7f1ce4002840 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 client.154142.objecter resend_mon_ops 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d0affd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6790/0 conn(0x7f1ce4002840 legacy=0x7f1ce4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670937703 0 0) 0x7f1ce4006590 con 0x7f1ce4002840 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 monclient(hunting): my global_id is 154145 2024-09-18T12:01:50.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d0c1461c0 con 0x7f1ce4002840 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3420317544 0 0) 0x7f1d0c1461c0 con 0x7f1ce4002840 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce4006590 con 0x7f1ce4002840 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 client.154142.objecter ms_handle_connect 0x7f1cf0038270 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1cfc004bc0 con 0x7f1ce4002840 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 monclient: dump: 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.156 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4149240208 0 0) 0x7f1ce4006590 con 0x7f1ce4002840 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ce400a140 con 0x7f1ce4002840 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f1cfc004f10 con 0x7f1ce4002840 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6790/0 conn(0x7f1ce4002840 legacy=0x7f1ce4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.136+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6790/0 conn(0x7f1ce4002840 legacy=0x7f1ce4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.144+0000 7f1d0bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.144+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1ce4009670 con 0x7f1ce40090e0 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 client.154142.objecter ms_handle_connect 0x7f1ce40090e0 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 client.154142.objecter resend_mon_ops 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0affd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6790/0 conn(0x7f1ce40090e0 legacy=0x7f1ce4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576672336 0 0) 0x7f1ce4009670 con 0x7f1ce40090e0 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 monclient(hunting): my global_id is 154148 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce4006590 con 0x7f1ce40090e0 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1794807557 0 0) 0x7f1ce4006590 con 0x7f1ce40090e0 2024-09-18T12:01:50.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce4009670 con 0x7f1ce40090e0 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f1cfc005530 con 0x7f1ce40090e0 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 monclient: dump: 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.161 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 243266600 0 0) 0x7f1ce4009670 con 0x7f1ce40090e0 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1ce4009930 con 0x7f1ce40090e0 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3831 (unknown 0 0 4032680568) 0x7f1cfc005840 con 0x7f1ce40090e0 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6790/0 conn(0x7f1ce40090e0 legacy=0x7f1ce4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d02ffd700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6790/0 conn(0x7f1ce40090e0 legacy=0x7f1ce4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:01:50.162 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 76, 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:50.163 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:50.164 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:01:50.165 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:50.166 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:01:50.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.168 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.169 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:01:50.169 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:01:50.169 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.169 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.169 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.170 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.172 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.173 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T12:01:50.174 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.174 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.174 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:01:50.174 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:01:50.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 10 librados: watch_flush enter 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 10 librados: watch_flush exit 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 20 client.154142.objecter shutdown clearing up homeless session... 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 10 client.154142.objecter successfully canceled tick 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.129:6812/20806 conn(0x7f1cf0038270 legacy=0x7f1cf003a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 10 monclient: shutdown 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 >> v1:172.21.15.112:6790/0 conn(0x7f1d0c099410 legacy=0x7f1d0c13fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0b7fe700 1 -- 172.21.15.112:0/2730910827 reap_dead start 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0b7fe700 1 -- 172.21.15.112:0/2730910827 reap_dead start 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 shutdown_connections 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 >> 172.21.15.112:0/2730910827 conn(0x7f1d0c0a1f90 msgr2=0x7f1d0c0976f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:50.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 shutdown_connections 2024-09-18T12:01:50.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 1 -- 172.21.15.112:0/2730910827 wait complete. 2024-09-18T12:01:50.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.148+0000 7f1d0bfff700 1 librados: shutdown 2024-09-18T12:01:50.177 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:01:50.177 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:01:50.177 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:01:50.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient: get_monmap_and_config 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient: build_initial_monmap 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient: monmap: 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:50.294133+0000 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:50.294133+0000 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:50.294 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 1 Processor -- start 2024-09-18T12:01:50.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 1 -- start start 2024-09-18T12:01:50.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient: init 2024-09-18T12:01:50.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:50.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:01:50.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient(hunting): picked mon.noname-c con 0x7f4d300a2040 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:50.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient(hunting): picked mon.noname-a con 0x7f4d3012edc0 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:50.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient(hunting): picked mon.noname-b con 0x7f4d3012f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:50.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300981a0 con 0x7f4d3012edc0 2024-09-18T12:01:50.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3006c120 con 0x7f4d300a2040 2024-09-18T12:01:50.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3006eef0 con 0x7f4d3012f8a0 2024-09-18T12:01:50.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:50.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d351ba700 10 monclient(hunting): authenticate will time out at 1882.135106s 2024-09-18T12:01:50.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d277fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f4d3012edc0 0x7f4d3012f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56498/0 (socket says 172.21.15.112:56498) 2024-09-18T12:01:50.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.288+0000 7f4d277fe700 1 -- 172.21.15.112:0/1067488896 learned_addr learned my addr 172.21.15.112:0/1067488896 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:50.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d277fe700 1 -- 172.21.15.112:0/1067488896 >> v1:172.21.15.112:6789/0 conn(0x7f4d3012edc0 legacy=0x7f4d3012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d349b9700 1 -- 172.21.15.112:0/1067488896 >> v1:172.21.15.112:6790/0 conn(0x7f4d3012f8a0 legacy=0x7f4d30137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504389377 0 0) 0x7f4d300981a0 con 0x7f4d3012edc0 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2ffff700 1 -- 172.21.15.112:0/1067488896 >> v1:172.21.15.129:6789/0 conn(0x7f4d300a2040 legacy=0x7f4d30097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): my global_id is 134349 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d10003570 con 0x7f4d3012edc0 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675979243 0 0) 0x7f4d3006eef0 con 0x7f4d3012f8a0 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): my global_id is 154151 2024-09-18T12:01:50.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d300981a0 con 0x7f4d3012f8a0 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679451266 0 0) 0x7f4d3006c120 con 0x7f4d300a2040 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient(hunting): my global_id is 144229 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d3006eef0 con 0x7f4d300a2040 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3637209164 0 0) 0x7f4d10003570 con 0x7f4d3012edc0 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 >> v1:172.21.15.112:6790/0 conn(0x7f4d3012f8a0 legacy=0x7f4d30137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 >> v1:172.21.15.129:6789/0 conn(0x7f4d300a2040 legacy=0x7f4d30097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 monclient: found mon.noname-a 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:50.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30096dd0 con 0x7f4d3012edc0 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: _finish_auth 0 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 5 monclient: authenticate success, global_id 134349 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4d20001940 con 0x7f4d3012edc0 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: dump: 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d20001bd0 con 0x7f4d3012edc0 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 4 set_mon_vals no callback set 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 1 -- 172.21.15.112:0/1067488896 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4d20004a90 con 0x7f4d3012edc0 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d2f7fe700 10 monclient: dump: 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 10 monclient: shutdown 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 1 -- 172.21.15.112:0/1067488896 >> v1:172.21.15.112:6789/0 conn(0x7f4d3012edc0 legacy=0x7f4d3012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 1 -- 172.21.15.112:0/1067488896 shutdown_connections 2024-09-18T12:01:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 1 -- 172.21.15.112:0/1067488896 >> 172.21.15.112:0/1067488896 conn(0x7f4d30067da0 msgr2=0x7f4d300a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 1 -- 172.21.15.112:0/1067488896 shutdown_connections 2024-09-18T12:01:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.292+0000 7f4d351ba700 1 -- 172.21.15.112:0/1067488896 wait complete. 2024-09-18T12:01:50.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient: build_initial_monmap 2024-09-18T12:01:50.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient: monmap: 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 librados: starting msgr at 2024-09-18T12:01:50.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 librados: starting objecter 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 Processor -- start 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 -- start start 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 librados: setting wanted keys 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 librados: calling monclient init 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient: init 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient(hunting): picked mon.noname-c con 0x7f4d300a2040 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient(hunting): picked mon.noname-a con 0x7f4d3012edc0 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient(hunting): picked mon.noname-b con 0x7f4d3012f8a0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3006f110 con 0x7f4d3012edc0 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d30134440 con 0x7f4d3012f8a0 2024-09-18T12:01:50.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300a0100 con 0x7f4d300a2040 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d351ba700 10 monclient(hunting): authenticate will time out at 1882.143171s 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d277fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f4d3012edc0 0x7f4d30132640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:56512/0 (socket says 172.21.15.112:56512) 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d277fe700 1 -- 172.21.15.112:0/1482815285 learned_addr learned my addr 172.21.15.112:0/1482815285 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.296+0000 7f4d2ffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f4d300a2040 0x7f4d30131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52952/0 (socket says 172.21.15.112:52952) 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2ffff700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.112:6790/0 conn(0x7f4d300a2040 legacy=0x7f4d30131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 client.?.objecter ms_handle_connect 0x7f4d300a2040 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 client.?.objecter ms_handle_connect 0x7f4d3012edc0 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d277fe700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.112:6789/0 conn(0x7f4d3012edc0 legacy=0x7f4d30132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721070875 0 0) 0x7f4d300a0100 con 0x7f4d300a2040 2024-09-18T12:01:50.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): my global_id is 154154 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d349b9700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6789/0 conn(0x7f4d3012f8a0 legacy=0x7f4d30149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d14004f10 con 0x7f4d300a2040 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 client.?.objecter ms_handle_connect 0x7f4d3012f8a0 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832123111 0 0) 0x7f4d3006f110 con 0x7f4d3012edc0 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): my global_id is 134352 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d300a0100 con 0x7f4d3012edc0 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802561542 0 0) 0x7f4d30134440 con 0x7f4d3012f8a0 2024-09-18T12:01:50.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): my global_id is 144232 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d3006f110 con 0x7f4d3012f8a0 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 269728279 0 0) 0x7f4d14004f10 con 0x7f4d300a2040 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d30134440 con 0x7f4d300a2040 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4d18001c00 con 0x7f4d300a2040 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): dump: 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.313 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3822120365 0 0) 0x7f4d300a0100 con 0x7f4d3012edc0 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d14004f10 con 0x7f4d3012edc0 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4d20001720 con 0x7f4d3012edc0 2024-09-18T12:01:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient(hunting): dump: 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 885316062 0 0) 0x7f4d30134440 con 0x7f4d300a2040 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6789/0 conn(0x7f4d3012f8a0 legacy=0x7f4d30149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.112:6789/0 conn(0x7f4d3012edc0 legacy=0x7f4d30132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient: _finish_hunting 0 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 monclient: found mon.c 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30132d50 con 0x7f4d300a2040 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient: _finish_auth 0 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:50.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 5 monclient: authenticate success, global_id 154154 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 10 monclient: _renew_subs 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d30134a80 con 0x7f4d300a2040 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 10 client.154154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 10 monclient: _renew_subs 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d30135100 con 0x7f4d300a2040 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 1 librados: init done 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d180022b0 con 0x7f4d300a2040 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4d18002600 con 0x7f4d300a2040 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2effd700 4 set_mon_vals no callback set 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d2cff9700 10 monclient: dump: 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.318 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.300+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d08006590 con 0x7f4d08002840 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f4d18002760 con 0x7f4d300a2040 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(691..691 src has 1..691) v4 ==== 5793+0+0 (unknown 1608782316 0 0) 0x7f4d18044040 con 0x7f4d300a2040 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 client.154154.objecter ms_dispatch 0x7f4d30000fd0 osd_map(691..691 src has 1..691) v4 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 3 client.154154.objecter handle_osd_map got epochs [691,691] > 0 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 3 client.154154.objecter handle_osd_map decoding full epoch 691 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 20 client.154154.objecter dump_active .. 0 homeless 2024-09-18T12:01:50.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d349b9700 10 monclient: get_auth_request con 0x7f4d140384b0 auth_method 0 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 client.154154.objecter ms_handle_connect 0x7f4d08002840 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 client.154154.objecter resend_mon_ops 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d277fe700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6789/0 conn(0x7f4d08002840 legacy=0x7f4d08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539979115 0 0) 0x7f4d08006590 con 0x7f4d08002840 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 monclient(hunting): my global_id is 144235 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30134440 con 0x7f4d08002840 2024-09-18T12:01:50.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1160846965 0 0) 0x7f4d30134440 con 0x7f4d08002840 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d08006590 con 0x7f4d08002840 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4d200019c0 con 0x7f4d08002840 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 monclient: dump: 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.322 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 10 client.154154.objecter ms_handle_connect 0x7f4d140384b0 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 702980908 0 0) 0x7f4d08006590 con 0x7f4d08002840 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.304+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d0800a140 con 0x7f4d08002840 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.308+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f4d20005030 con 0x7f4d08002840 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.308+0000 7f4d2cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.308+0000 7f4d2cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.308+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6789/0 conn(0x7f4d08002840 legacy=0x7f4d08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.323 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.308+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6789/0 conn(0x7f4d08002840 legacy=0x7f4d08004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:50.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.316+0000 7f4d351ba700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:01:50.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.316+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d08009670 con 0x7f4d080090e0 2024-09-18T12:01:50.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d277fe700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6789/0 conn(0x7f4d080090e0 legacy=0x7f4d08008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 10 client.154154.objecter ms_handle_connect 0x7f4d080090e0 2024-09-18T12:01:50.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 10 client.154154.objecter resend_mon_ops 2024-09-18T12:01:50.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335934984 0 0) 0x7f4d08009670 con 0x7f4d080090e0 2024-09-18T12:01:50.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 10 monclient(hunting): my global_id is 144238 2024-09-18T12:01:50.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d08006590 con 0x7f4d080090e0 2024-09-18T12:01:50.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2393827727 0 0) 0x7f4d08006590 con 0x7f4d080090e0 2024-09-18T12:01:50.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d08009670 con 0x7f4d080090e0 2024-09-18T12:01:50.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f4d20005690 con 0x7f4d080090e0 2024-09-18T12:01:50.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:50.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.320+0000 7f4d2cff9700 10 monclient: dump: 2024-09-18T12:01:50.327 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2952052120 0 0) 0x7f4d08009670 con 0x7f4d080090e0 2024-09-18T12:01:50.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d08009930 con 0x7f4d080090e0 2024-09-18T12:01:50.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 1553045316) 0x7f4d20005aa0 con 0x7f4d080090e0 2024-09-18T12:01:50.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d2cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:01:50.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d2cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:50.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6789/0 conn(0x7f4d080090e0 legacy=0x7f4d08008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d2cff9700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6789/0 conn(0x7f4d080090e0 legacy=0x7f4d08008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:01:50.329 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:01:50.329 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 76, 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 2, 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:01:50.330 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:50.331 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:01:50.332 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:01:50.333 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:01:50.334 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.335 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.336 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.337 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:01:50.338 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:01:50.338 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:01:50.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.338 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:01:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:01:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:01:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:01:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:01:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T12:01:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:01:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:01:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:01:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:01:50.344 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:01:50.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 10 librados: watch_flush enter 2024-09-18T12:01:50.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 10 librados: watch_flush exit 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 20 client.154154.objecter shutdown clearing up homeless session... 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 10 client.154154.objecter successfully canceled tick 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.129:6812/20806 conn(0x7f4d140384b0 legacy=0x7f4d1403a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 10 monclient: shutdown 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 >> v1:172.21.15.112:6790/0 conn(0x7f4d300a2040 legacy=0x7f4d30131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d349b9700 1 -- 172.21.15.112:0/1482815285 reap_dead start 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d349b9700 1 -- 172.21.15.112:0/1482815285 reap_dead start 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 shutdown_connections 2024-09-18T12:01:50.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 >> 172.21.15.112:0/1482815285 conn(0x7f4d30067da0 msgr2=0x7f4d3006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:50.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 shutdown_connections 2024-09-18T12:01:50.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 1 -- 172.21.15.112:0/1482815285 wait complete. 2024-09-18T12:01:50.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.324+0000 7f4d351ba700 1 librados: shutdown 2024-09-18T12:01:50.348 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:01:50.348 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T12:01:50.348 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:01:50.348 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T12:01:50.348 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:01:50.349 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:01:50.349 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:01:50.370 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:01:50.370 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:01:50.370 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:01:50.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient: get_monmap_and_config 2024-09-18T12:01:50.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient: build_initial_monmap 2024-09-18T12:01:50.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient: monmap: 2024-09-18T12:01:50.521 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:50.521 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.522 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:50.522768+0000 2024-09-18T12:01:50.522 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:50.522768+0000 2024-09-18T12:01:50.522 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:50.522 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.522 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:50.522 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:50.522 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:50.522 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 1 Processor -- start 2024-09-18T12:01:50.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 1 -- start start 2024-09-18T12:01:50.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient: init 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient(hunting): picked mon.noname-a con 0x7ff3f40a0570 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient(hunting): picked mon.noname-c con 0x7ff3f40a1050 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient(hunting): picked mon.noname-b con 0x7ff3f4097a50 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f4097070 con 0x7ff3f40a0570 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f4096d90 con 0x7ff3f40a1050 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f4097e50 con 0x7ff3f4097a50 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3fb837700 10 monclient(hunting): authenticate will time out at 1882.363798s 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.516+0000 7ff3f95d3700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7ff3f40a0570 legacy=0x7ff3f40a0940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3f9dd4700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff3f4097a50 0x7ff3f413aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52966/0 (socket says 172.21.15.112:52966) 2024-09-18T12:01:50.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3f9dd4700 1 -- 172.21.15.112:0/3266600758 learned_addr learned my addr 172.21.15.112:0/3266600758 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3f9dd4700 1 -- 172.21.15.112:0/3266600758 >> v1:172.21.15.112:6790/0 conn(0x7ff3f4097a50 legacy=0x7ff3f413aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3f8dd2700 1 -- 172.21.15.112:0/3266600758 >> v1:172.21.15.129:6789/0 conn(0x7ff3f40a1050 legacy=0x7ff3f4137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84150263 0 0) 0x7ff3f4097e50 con 0x7ff3f4097a50 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient(hunting): my global_id is 154157 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e0003570 con 0x7ff3f4097a50 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688643938 0 0) 0x7ff3f4096d90 con 0x7ff3f40a1050 2024-09-18T12:01:50.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient(hunting): my global_id is 144241 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f4097e50 con 0x7ff3f40a1050 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428197298 0 0) 0x7ff3e0003570 con 0x7ff3f4097a50 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 >> v1:172.21.15.129:6789/0 conn(0x7ff3f40a1050 legacy=0x7ff3f4137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 >> v1:172.21.15.112:6789/0 conn(0x7ff3f40a0570 legacy=0x7ff3f40a0940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: _finish_hunting 0 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 monclient: found mon.noname-b 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f40a2050 con 0x7ff3f4097a50 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: _finish_auth 0 2024-09-18T12:01:50.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3ec001d80 con 0x7ff3f4097a50 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 5 monclient: authenticate success, global_id 154157 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: dump: 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.528 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ec001f60 con 0x7ff3f4097a50 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 4 set_mon_vals no callback set 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 1 -- 172.21.15.112:0/3266600758 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3ec004e10 con 0x7ff3f4097a50 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3ebfff700 10 monclient: dump: 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.529 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.530 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.530 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 10 monclient: shutdown 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 1 -- 172.21.15.112:0/3266600758 >> v1:172.21.15.112:6790/0 conn(0x7ff3f4097a50 legacy=0x7ff3f413aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 1 -- 172.21.15.112:0/3266600758 shutdown_connections 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.520+0000 7ff3fb837700 1 -- 172.21.15.112:0/3266600758 >> 172.21.15.112:0/3266600758 conn(0x7ff3f4067680 msgr2=0x7ff3f4067a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 -- 172.21.15.112:0/3266600758 shutdown_connections 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 -- 172.21.15.112:0/3266600758 wait complete. 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient: build_initial_monmap 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient: monmap: 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:50.534 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 librados: starting msgr at 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 librados: starting objecter 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 Processor -- start 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 -- start start 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 librados: setting wanted keys 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 librados: calling monclient init 2024-09-18T12:01:50.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient: init 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient(hunting): picked mon.noname-b con 0x7ff3f4097a50 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient(hunting): picked mon.noname-a con 0x7ff3f40a0570 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient(hunting): picked mon.noname-c con 0x7ff3f40a1050 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f4122c90 con 0x7ff3f40a0570 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f413dd40 con 0x7ff3f4097a50 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f4120ca0 con 0x7ff3f40a1050 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3fb837700 10 monclient(hunting): authenticate will time out at 1882.370236s 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3f8dd2700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7ff3f40a0570 legacy=0x7ff3f4146880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3f9dd4700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff3f40a1050 0x7ff3f4149fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52974/0 (socket says 172.21.15.112:52974) 2024-09-18T12:01:50.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.524+0000 7ff3f9dd4700 1 -- 172.21.15.112:0/2499382996 learned_addr learned my addr 172.21.15.112:0/2499382996 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3f9dd4700 1 -- 172.21.15.112:0/2499382996 >> v1:172.21.15.112:6790/0 conn(0x7ff3f40a1050 legacy=0x7ff3f4149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 client.?.objecter ms_handle_connect 0x7ff3f40a1050 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 client.?.objecter ms_handle_connect 0x7ff3f4097a50 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3f95d3700 1 -- 172.21.15.112:0/2499382996 >> v1:172.21.15.129:6789/0 conn(0x7ff3f4097a50 legacy=0x7ff3f413d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139551375 0 0) 0x7ff3f4120ca0 con 0x7ff3f40a1050 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): my global_id is 154160 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8004f10 con 0x7ff3f40a1050 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084597682 0 0) 0x7ff3f413dd40 con 0x7ff3f4097a50 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:50.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): my global_id is 144244 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f4120ca0 con 0x7ff3f4097a50 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2397059321 0 0) 0x7ff3d8004f10 con 0x7ff3f40a1050 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f413dd40 con 0x7ff3f40a1050 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3ec004bb0 con 0x7ff3f40a1050 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): dump: 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2318805046 0 0) 0x7ff3f413dd40 con 0x7ff3f40a1050 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 >> v1:172.21.15.129:6789/0 conn(0x7ff3f4097a50 legacy=0x7ff3f413d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 >> v1:172.21.15.112:6789/0 conn(0x7ff3f40a0570 legacy=0x7ff3f4146880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient: _finish_hunting 0 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 monclient: found mon.c 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f4120440 con 0x7ff3f40a1050 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient: _finish_auth 0 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 5 monclient: authenticate success, global_id 154160 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 10 monclient: _renew_subs 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f41226d0 con 0x7ff3f40a1050 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 10 client.154160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 10 monclient: _renew_subs 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f413eb30 con 0x7ff3f40a1050 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 1 librados: init done 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ec005140 con 0x7ff3f40a1050 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:50.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff3ec0054d0 con 0x7ff3f40a1050 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3eb7fe700 4 set_mon_vals no callback set 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 10 monclient: dump: 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:50.541 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:50.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d40050b0 con 0x7ff3f40a1050 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.528+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ff3ec0056b0 con 0x7ff3f40a1050 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.532+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(691..691 src has 1..691) v4 ==== 5793+0+0 (unknown 1608782316 0 0) 0x7ff3ec0445c0 con 0x7ff3f40a1050 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.532+0000 7ff3e97fa700 10 client.154160.objecter ms_dispatch 0x7ff3f4000fd0 osd_map(691..691 src has 1..691) v4 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.532+0000 7ff3e97fa700 3 client.154160.objecter handle_osd_map got epochs [691,691] > 0 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.532+0000 7ff3e97fa700 3 client.154160.objecter handle_osd_map decoding full epoch 691 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.532+0000 7ff3e97fa700 20 client.154160.objecter dump_active .. 0 homeless 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.532+0000 7ff3f95d3700 10 monclient: get_auth_request con 0x7ff3d8037f50 auth_method 0 2024-09-18T12:01:50.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.532+0000 7ff3e97fa700 10 client.154160.objecter ms_handle_connect 0x7ff3d8037f50 2024-09-18T12:01:50.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.540+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7ff3ec0150c0 con 0x7ff3f40a1050 2024-09-18T12:01:50.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.540+0000 7ff3e97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:50.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.540+0000 7ff3e97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:50.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.756+0000 7ff3fb837700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:01:50.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.756+0000 7ff3fb837700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:01:50.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.756+0000 7ff3fb837700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:50.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.756+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff3d40058e0 con 0x7ff3f40a1050 2024-09-18T12:01:50.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.756+0000 7ff3e97fa700 1 -- 172.21.15.112:0/2499382996 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2125273252) 0x7ff3ec014ac0 con 0x7ff3f40a1050 2024-09-18T12:01:50.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.756+0000 7ff3e97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:01:50.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.756+0000 7ff3e97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:50.763 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:01:50.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 10 librados: watch_flush enter 2024-09-18T12:01:50.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 10 librados: watch_flush exit 2024-09-18T12:01:50.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 20 client.154160.objecter shutdown clearing up homeless session... 2024-09-18T12:01:50.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 10 client.154160.objecter successfully canceled tick 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 >> v1:172.21.15.129:6812/20806 conn(0x7ff3d8037f50 legacy=0x7ff3d803a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 10 monclient: shutdown 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 >> v1:172.21.15.112:6790/0 conn(0x7ff3f40a1050 legacy=0x7ff3f4149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 shutdown_connections 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 >> 172.21.15.112:0/2499382996 conn(0x7ff3f4067680 msgr2=0x7ff3f40985e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 shutdown_connections 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 1 -- 172.21.15.112:0/2499382996 wait complete. 2024-09-18T12:01:50.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:50.760+0000 7ff3fb837700 1 librados: shutdown 2024-09-18T12:01:53.778 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:01:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.884+0000 7f5aab59e700 10 monclient: get_monmap_and_config 2024-09-18T12:01:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.884+0000 7f5aab59e700 10 monclient: build_initial_monmap 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.884+0000 7f5aab59e700 10 monclient: monmap: 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:53.891961+0000 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:53.891961+0000 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:53.892 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 1 Processor -- start 2024-09-18T12:01:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 1 -- start start 2024-09-18T12:01:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 10 monclient: init 2024-09-18T12:01:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:01:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 10 monclient(hunting): picked mon.noname-c con 0x7f5aac0a15f0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 10 monclient(hunting): picked mon.noname-b con 0x7f5aac0a19c0 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 10 monclient(hunting): picked mon.noname-a con 0x7f5aac138530 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac0a3250 con 0x7f5aac138530 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac064370 con 0x7f5aac0a15f0 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac06a5a0 con 0x7f5aac0a19c0 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aab59e700 10 monclient(hunting): authenticate will time out at 1885.732920s 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aaad9d700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f5aac138530 legacy=0x7f5aac13a910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa1d9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5aac0a19c0 0x7f5aac0a1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:52990/0 (socket says 172.21.15.112:52990) 2024-09-18T12:01:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa1d9b700 1 -- 172.21.15.112:0/2481674080 learned_addr learned my addr 172.21.15.112:0/2481674080 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa1d9b700 1 -- 172.21.15.112:0/2481674080 >> v1:172.21.15.112:6790/0 conn(0x7f5aac0a19c0 legacy=0x7f5aac0a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aaa59c700 1 -- 172.21.15.112:0/2481674080 >> v1:172.21.15.129:6789/0 conn(0x7f5aac0a15f0 legacy=0x7f5aac06c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336237227 0 0) 0x7f5aac06a5a0 con 0x7f5aac0a19c0 2024-09-18T12:01:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 10 monclient(hunting): my global_id is 154163 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a98003570 con 0x7f5aac0a19c0 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671674370 0 0) 0x7f5aac064370 con 0x7f5aac0a15f0 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 10 monclient(hunting): my global_id is 144247 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.888+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aac06a5a0 con 0x7f5aac0a15f0 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1876977175 0 0) 0x7f5aac06a5a0 con 0x7f5aac0a15f0 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 >> v1:172.21.15.112:6790/0 conn(0x7f5aac0a19c0 legacy=0x7f5aac0a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 >> v1:172.21.15.112:6789/0 conn(0x7f5aac138530 legacy=0x7f5aac13a910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:01:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: _finish_hunting 0 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 1 monclient: found mon.noname-c 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac06b590 con 0x7f5aac0a15f0 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: _finish_auth 0 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 5 monclient: authenticate success, global_id 144247 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5a9c001a30 con 0x7f5aac0a15f0 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: dump: 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:53.903 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a9c001c10 con 0x7f5aac0a15f0 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 10 monclient: get_monmap_and_config success 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 1 -- 172.21.15.112:0/2481674080 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5a9c0020a0 con 0x7f5aac0a15f0 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 4 set_mon_vals no callback set 2024-09-18T12:01:53.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aa9d9b700 10 monclient: dump: 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:53.905 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 10 monclient: shutdown 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 -- 172.21.15.112:0/2481674080 >> v1:172.21.15.129:6789/0 conn(0x7f5aac0a15f0 legacy=0x7f5aac06c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 -- 172.21.15.112:0/2481674080 shutdown_connections 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 -- 172.21.15.112:0/2481674080 >> 172.21.15.112:0/2481674080 conn(0x7f5aac0a0da0 msgr2=0x7f5aac067680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 -- 172.21.15.112:0/2481674080 shutdown_connections 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 -- 172.21.15.112:0/2481674080 wait complete. 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 10 monclient: build_initial_monmap 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 10 monclient: monmap: 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:53.906 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 librados: starting msgr at 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 librados: starting objecter 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 Processor -- start 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 -- start start 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 librados: setting wanted keys 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 1 librados: calling monclient init 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.892+0000 7f5aab59e700 10 monclient: init 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:01:53.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient(hunting): picked mon.noname-b con 0x7f5aac0a15f0 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient(hunting): picked mon.noname-a con 0x7f5aac0a19c0 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient(hunting): picked mon.noname-c con 0x7f5aac138530 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac06f000 con 0x7f5aac0a19c0 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac06e5e0 con 0x7f5aac0a15f0 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac137b80 con 0x7f5aac138530 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa1d9b700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f5aac0a19c0 legacy=0x7f5aac1322e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient(hunting): authenticate will time out at 1885.740257s 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aaad9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5aac138530 0x7f5aac149670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:53002/0 (socket says 172.21.15.112:53002) 2024-09-18T12:01:53.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aaad9d700 1 -- 172.21.15.112:0/2674496414 learned_addr learned my addr 172.21.15.112:0/2674496414 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:53.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aaad9d700 1 -- 172.21.15.112:0/2674496414 >> v1:172.21.15.112:6790/0 conn(0x7f5aac138530 legacy=0x7f5aac149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:53.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 client.?.objecter ms_handle_connect 0x7f5aac138530 2024-09-18T12:01:53.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:53.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aaa59c700 1 -- 172.21.15.112:0/2674496414 >> v1:172.21.15.129:6789/0 conn(0x7f5aac0a15f0 legacy=0x7f5aac141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:53.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 client.?.objecter ms_handle_connect 0x7f5aac0a15f0 2024-09-18T12:01:53.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:01:53.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484198263 0 0) 0x7f5aac137b80 con 0x7f5aac138530 2024-09-18T12:01:53.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): my global_id is 154166 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a8c004f10 con 0x7f5aac138530 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143885086 0 0) 0x7f5aac06e5e0 con 0x7f5aac0a15f0 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): my global_id is 144250 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aac137b80 con 0x7f5aac0a15f0 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774443773 0 0) 0x7f5a8c004f10 con 0x7f5aac138530 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aac06e5e0 con 0x7f5aac138530 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5aa4004800 con 0x7f5aac138530 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:53.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): dump: 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:01:53.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1116798927 0 0) 0x7f5aac137b80 con 0x7f5aac0a15f0 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a8c004f10 con 0x7f5aac0a15f0 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5a9c001970 con 0x7f5aac0a15f0 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient(hunting): dump: 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:53.916 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1990134343 0 0) 0x7f5aac06e5e0 con 0x7f5aac138530 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 >> v1:172.21.15.129:6789/0 conn(0x7f5aac0a15f0 legacy=0x7f5aac141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 >> v1:172.21.15.112:6789/0 conn(0x7f5aac0a19c0 legacy=0x7f5aac1322e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 monclient: found mon.c 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac135ce0 con 0x7f5aac138530 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 10 monclient: _finish_auth 0 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aa2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:01:53.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 5 monclient: authenticate success, global_id 154166 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient: _renew_subs 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac137810 con 0x7f5aac138530 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 client.154166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient: _renew_subs 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac1368b0 con 0x7f5aac138530 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.896+0000 7f5aab59e700 1 librados: init done 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aa40049e0 con 0x7f5aac138530 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5aa4004d30 con 0x7f5aac138530 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:01:53.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa8d99700 4 set_mon_vals no callback set 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 10 monclient: dump: 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5aa4004fc0 con 0x7f5aac138530 2024-09-18T12:01:53.919 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aab59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aab59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a900050b0 con 0x7f5aac138530 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(691..691 src has 1..691) v4 ==== 5793+0+0 (unknown 1608782316 0 0) 0x7f5aa40442a0 con 0x7f5aac138530 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 10 client.154166.objecter ms_dispatch 0x7f5aac000fd0 osd_map(691..691 src has 1..691) v4 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 3 client.154166.objecter handle_osd_map got epochs [691,691] > 0 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 3 client.154166.objecter handle_osd_map decoding full epoch 691 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aa2ffd700 20 client.154166.objecter dump_active .. 0 homeless 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.900+0000 7f5aaa59c700 10 monclient: get_auth_request con 0x7f5a8c038210 auth_method 0 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.904+0000 7f5aa2ffd700 10 client.154166.objecter ms_handle_connect 0x7f5a8c038210 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.908+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f5aa4014ce0 con 0x7f5aac138530 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.908+0000 7f5aa2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:01:53.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:53.908+0000 7f5aa2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:01:54.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.128+0000 7f5aab59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:01:54.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.128+0000 7f5aab59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:01:54.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.128+0000 7f5aab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:01:54.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.128+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5a900058e0 con 0x7f5aac138530 2024-09-18T12:01:54.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.128+0000 7f5aa2ffd700 1 -- 172.21.15.112:0/2674496414 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1565689261) 0x7f5aa40146e0 con 0x7f5aac138530 2024-09-18T12:01:54.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.128+0000 7f5aa2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:01:54.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.128+0000 7f5aa2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:01:54.134 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:01:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 10 librados: watch_flush enter 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 10 librados: watch_flush exit 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 20 client.154166.objecter shutdown clearing up homeless session... 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 10 client.154166.objecter successfully canceled tick 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 >> v1:172.21.15.129:6812/20806 conn(0x7f5a8c038210 legacy=0x7f5a8c03a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 10 monclient: shutdown 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 >> v1:172.21.15.112:6790/0 conn(0x7f5aac138530 legacy=0x7f5aac149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 shutdown_connections 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 >> 172.21.15.112:0/2674496414 conn(0x7f5aac0a0da0 msgr2=0x7f5aac13a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 shutdown_connections 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 1 -- 172.21.15.112:0/2674496414 wait complete. 2024-09-18T12:01:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:54.132+0000 7f5aab59e700 1 librados: shutdown 2024-09-18T12:01:57.149 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:01:57.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient: get_monmap_and_config 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient: build_initial_monmap 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient: monmap: 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:01:57.260998+0000 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:01:57.260998+0000 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:01:57.260 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 1 Processor -- start 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 1 -- start start 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient: init 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient: _reopen_session rank -1 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient(hunting): picked mon.noname-b con 0x7fa7b4067680 addr v1:172.21.15.112:6790/0 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient(hunting): picked mon.noname-c con 0x7fa7b4098660 addr v1:172.21.15.129:6789/0 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient(hunting): picked mon.noname-a con 0x7fa7b4098a60 addr v1:172.21.15.112:6789/0 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b4067a50 con 0x7fa7b4098a60 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b4096b70 con 0x7fa7b4098660 2024-09-18T12:01:57.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b409e130 con 0x7fa7b4067680 2024-09-18T12:01:57.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient(hunting): _renew_subs 2024-09-18T12:01:57.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7bb0cb700 10 monclient(hunting): authenticate will time out at 1889.102712s 2024-09-18T12:01:57.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7ba8ca700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa7b413a900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:01:57.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7ba0c9700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa7b4067680 0x7fa7b412ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:53018/0 (socket says 172.21.15.112:53018) 2024-09-18T12:01:57.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.260+0000 7fa7ba0c9700 1 -- 172.21.15.112:0/3615302383 learned_addr learned my addr 172.21.15.112:0/3615302383 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:01:57.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.264+0000 7fa7ba0c9700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6790/0 conn(0x7fa7b4067680 legacy=0x7fa7b412ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:57.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.264+0000 7fa7b98c8700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.129:6789/0 conn(0x7fa7b4098660 legacy=0x7fa7b41371d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:01:57.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.464+0000 7fa7ba8ca700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa7b413a900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:01:57.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:57.868+0000 7fa7ba8ca700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa7b413a900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:01:58.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:01:58.672+0000 7fa7ba8ca700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa7b413a900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:00.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 10 monclient(hunting): tick 2024-09-18T12:02:00.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 1 monclient(hunting): continuing hunt 2024-09-18T12:02:00.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:02:00.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6790/0 conn(0x7fa7b4067680 legacy=0x7fa7b412ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:00.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.129:6789/0 conn(0x7fa7b4098660 legacy=0x7fa7b41371d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:00.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa7b413a900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 10 monclient(hunting): picked mon.noname-a con 0x7fa7b4098a60 addr v1:172.21.15.112:6789/0 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 10 monclient(hunting): picked mon.noname-b con 0x7fa798006fa0 addr v1:172.21.15.112:6790/0 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 10 monclient(hunting): picked mon.noname-c con 0x7fa7b4098660 addr v1:172.21.15.129:6789/0 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b406f9c0 con 0x7fa7b4098a60 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b4067a50 con 0x7fa7b4098660 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa798010560 con 0x7fa798006fa0 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7abfff700 10 monclient(hunting): _renew_subs 2024-09-18T12:02:00.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.260+0000 7fa7ba0c9700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:00.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.264+0000 7fa7b98c8700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6790/0 conn(0x7fa798006fa0 legacy=0x7fa798009400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:00.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.264+0000 7fa7ba8ca700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.129:6789/0 conn(0x7fa7b4098660 legacy=0x7fa79800cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:00.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.464+0000 7fa7ba0c9700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:00.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:00.868+0000 7fa7ba0c9700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:01.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:01.672+0000 7fa7ba0c9700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:03.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 10 monclient(hunting): tick 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 1 monclient(hunting): continuing hunt 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6790/0 conn(0x7fa798006fa0 legacy=0x7fa798009400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.129:6789/0 conn(0x7fa7b4098660 legacy=0x7fa79800cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 10 monclient(hunting): picked mon.noname-c con 0x7fa79800f640 addr v1:172.21.15.129:6789/0 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 10 monclient(hunting): picked mon.noname-a con 0x7fa7b4098a60 addr v1:172.21.15.112:6789/0 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 10 monclient(hunting): picked mon.noname-b con 0x7fa7980106f0 addr v1:172.21.15.112:6790/0 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa798011820 con 0x7fa7b4098a60 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b406f9c0 con 0x7fa79800f640 2024-09-18T12:02:03.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 1 -- 172.21.15.112:0/3615302383 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7980112c0 con 0x7fa7980106f0 2024-09-18T12:02:03.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7abfff700 10 monclient(hunting): _renew_subs 2024-09-18T12:02:03.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.260+0000 7fa7b98c8700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.264+0000 7fa7ba8ca700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6790/0 conn(0x7fa7980106f0 legacy=0x7fa798010b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.264+0000 7fa7ba0c9700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.129:6789/0 conn(0x7fa79800f640 legacy=0x7fa79800fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:03.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.464+0000 7fa7b98c8700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:03.868+0000 7fa7b98c8700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:04.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.672+0000 7fa7b98c8700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:04.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428347010 0 0) 0x7fa7b406f9c0 con 0x7fa79800f640 2024-09-18T12:02:04.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:04.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:04.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient(hunting): my global_id is 144253 2024-09-18T12:02:04.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a0003570 con 0x7fa79800f640 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520652724 0 0) 0x7fa7a0003570 con 0x7fa79800f640 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6790/0 conn(0x7fa7980106f0 legacy=0x7fa798010b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.112:6789/0 conn(0x7fa7b4098a60 legacy=0x7fa798005860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient: _finish_hunting 0 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 1 monclient: found mon.noname-c 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7980114f0 con 0x7fa79800f640 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient: _finish_auth 0 2024-09-18T12:02:04.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa7b00018e0 con 0x7fa79800f640 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7bb0cb700 5 monclient: authenticate success, global_id 144253 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7bb0cb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7b90c7700 10 monclient: dump: 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:04.781 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:04.782 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:04.782 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:04.782 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:04.782 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:04.782 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:04.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.776+0000 7fa7bb0cb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:02:04.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7b0001b40 con 0x7fa79800f640 2024-09-18T12:02:04.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7b90c7700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:02:04.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7b90c7700 1 -- 172.21.15.112:0/3615302383 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa7b00048f0 con 0x7fa79800f640 2024-09-18T12:02:04.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 10 monclient: get_monmap_and_config success 2024-09-18T12:02:04.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 4 set_mon_vals no callback set 2024-09-18T12:02:04.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7b90c7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:04.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7b90c7700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:02:04.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7b90c7700 10 monclient: dump: 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:04.787 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:04.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 10 monclient: shutdown 2024-09-18T12:02:04.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:02:04.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/3615302383 >> v1:172.21.15.129:6789/0 conn(0x7fa79800f640 legacy=0x7fa79800fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/3615302383 shutdown_connections 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/3615302383 >> 172.21.15.112:0/3615302383 conn(0x7fa7b40a1ec0 msgr2=0x7fa7b40a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/3615302383 shutdown_connections 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/3615302383 wait complete. 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 10 monclient: build_initial_monmap 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.780+0000 7fa7bb0cb700 10 monclient: monmap: 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:02:04.789 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 librados: starting msgr at 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 librados: starting objecter 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 Processor -- start 2024-09-18T12:02:04.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 -- start start 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 librados: setting wanted keys 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 librados: calling monclient init 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 10 monclient: init 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 10 monclient: _reopen_session rank -1 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 10 monclient(hunting): picked mon.noname-a con 0x7fa79800f640 addr v1:172.21.15.112:6789/0 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 10 monclient(hunting): picked mon.noname-b con 0x7fa7980106f0 addr v1:172.21.15.129:6789/0 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 10 monclient(hunting): picked mon.noname-c con 0x7fa7b4098a60 addr v1:172.21.15.112:6790/0 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b409e340 con 0x7fa79800f640 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b412e7c0 con 0x7fa7980106f0 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b411c390 con 0x7fa7b4098a60 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 10 monclient(hunting): _renew_subs 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7ba0c9700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7fa79800f640 legacy=0x7fa7b4118e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:04.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7bb0cb700 10 monclient(hunting): authenticate will time out at 1896.626153s 2024-09-18T12:02:04.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7ba8ca700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa7b4098a60 0x7fa7b4119c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39932/0 (socket says 172.21.15.112:39932) 2024-09-18T12:02:04.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7b98c8700 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa7980106f0 0x7fa7b4119510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.112:58284/0 (socket says 172.21.15.112:58284) 2024-09-18T12:02:04.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.784+0000 7fa7ba8ca700 1 -- 172.21.15.112:0/822772276 learned_addr learned my addr 172.21.15.112:0/822772276 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:02:04.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7b98c8700 1 -- 172.21.15.112:0/822772276 >> v1:172.21.15.129:6789/0 conn(0x7fa7980106f0 legacy=0x7fa7b4119510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:04.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 client.?.objecter ms_handle_connect 0x7fa7980106f0 2024-09-18T12:02:04.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:02:04.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 client.?.objecter ms_handle_connect 0x7fa7b4098a60 2024-09-18T12:02:04.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:02:04.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7ba8ca700 1 -- 172.21.15.112:0/822772276 >> v1:172.21.15.112:6790/0 conn(0x7fa7b4098a60 legacy=0x7fa7b4119c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:04.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134794330 0 0) 0x7fa7b412e7c0 con 0x7fa7980106f0 2024-09-18T12:02:04.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient(hunting): my global_id is 144256 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa798010240 con 0x7fa7980106f0 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1085469138 0 0) 0x7fa798010560 con 0x7fa7980106f0 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7b412e7c0 con 0x7fa7980106f0 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa7a4001960 con 0x7fa7980106f0 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient(hunting): dump: 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:04.797 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 151004744 0 0) 0x7fa7b412e7c0 con 0x7fa7980106f0 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 >> v1:172.21.15.112:6790/0 conn(0x7fa7b4098a60 legacy=0x7fa7b4119c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 >> v1:172.21.15.112:6789/0 conn(0x7fa79800f640 legacy=0x7fa7b4118e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 monclient: found mon.b 2024-09-18T12:02:04.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b412c3b0 con 0x7fa7980106f0 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient: _finish_auth 0 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 5 monclient: authenticate success, global_id 144256 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 10 monclient: _renew_subs 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b411b430 con 0x7fa7980106f0 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 10 client.144256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 10 monclient: _renew_subs 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b414b1c0 con 0x7fa7980106f0 2024-09-18T12:02:04.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7bb0cb700 1 librados: init done 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7a4001ce0 con 0x7fa7980106f0 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa7a4002450 con 0x7fa7980106f0 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7b88c6700 4 set_mon_vals no callback set 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.788+0000 7fa7aa7fc700 10 monclient: dump: 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:04.800 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa7a40025b0 con 0x7fa7980106f0 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7bb0cb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7bb0cb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7bb0cb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79c0050b0 con 0x7fa7980106f0 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(691..691 src has 1..691) v4 ==== 5793+0+0 (unknown 1608782316 0 0) 0x7fa7a4043d70 con 0x7fa7980106f0 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7aa7fc700 10 client.144256.objecter ms_dispatch 0x7fa7b4000fd0 osd_map(691..691 src has 1..691) v4 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7aa7fc700 3 client.144256.objecter handle_osd_map got epochs [691,691] > 0 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7aa7fc700 3 client.144256.objecter handle_osd_map decoding full epoch 691 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7ba0c9700 10 monclient: get_auth_request con 0x7fa798039730 auth_method 0 2024-09-18T12:02:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.792+0000 7fa7aa7fc700 20 client.144256.objecter dump_active .. 0 homeless 2024-09-18T12:02:04.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.796+0000 7fa7aa7fc700 10 client.144256.objecter ms_handle_connect 0x7fa798039730 2024-09-18T12:02:04.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.804+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fa7a406e670 con 0x7fa7980106f0 2024-09-18T12:02:04.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.804+0000 7fa7aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:02:04.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:04.804+0000 7fa7aa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:02:05.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.048+0000 7fa7bb0cb700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:02:05.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.048+0000 7fa7bb0cb700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:02:05.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.048+0000 7fa7bb0cb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-09-18T12:02:05.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.048+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa79c0058e0 con 0x7fa7980106f0 2024-09-18T12:02:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.048+0000 7fa7aa7fc700 1 -- 172.21.15.112:0/822772276 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2309906271) 0x7fa7a4002860 con 0x7fa7980106f0 2024-09-18T12:02:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.048+0000 7fa7aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:02:05.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.048+0000 7fa7aa7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:02:05.049 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:02:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 10 librados: watch_flush enter 2024-09-18T12:02:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 10 librados: watch_flush exit 2024-09-18T12:02:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 20 client.144256.objecter shutdown clearing up homeless session... 2024-09-18T12:02:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 10 client.144256.objecter successfully canceled tick 2024-09-18T12:02:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 >> v1:172.21.15.129:6812/20806 conn(0x7fa798039730 legacy=0x7fa79803bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 10 monclient: shutdown 2024-09-18T12:02:05.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:02:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 >> v1:172.21.15.129:6789/0 conn(0x7fa7980106f0 legacy=0x7fa7b4119510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 shutdown_connections 2024-09-18T12:02:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 >> 172.21.15.112:0/822772276 conn(0x7fa7b40a1ec0 msgr2=0x7fa7b412c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:02:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 shutdown_connections 2024-09-18T12:02:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 1 -- 172.21.15.112:0/822772276 wait complete. 2024-09-18T12:02:05.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.052+0000 7fa7bb0cb700 1 librados: shutdown 2024-09-18T12:02:05.062 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:02:05.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:02:05.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:02:05.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient: get_monmap_and_config 2024-09-18T12:02:05.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient: build_initial_monmap 2024-09-18T12:02:05.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient: monmap: 2024-09-18T12:02:05.168 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:02:05.168 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.169 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:02:05.168905+0000 2024-09-18T12:02:05.169 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:02:05.168905+0000 2024-09-18T12:02:05.169 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:02:05.169 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.169 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:02:05.169 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:02:05.170 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:02:05.170 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 1 Processor -- start 2024-09-18T12:02:05.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 1 -- start start 2024-09-18T12:02:05.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient: init 2024-09-18T12:02:05.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient: _reopen_session rank -1 2024-09-18T12:02:05.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:02:05.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient(hunting): picked mon.noname-b con 0x7f64600a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:02:05.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient(hunting): picked mon.noname-a con 0x7f64600a0750 addr v1:172.21.15.112:6789/0 2024-09-18T12:02:05.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient(hunting): picked mon.noname-c con 0x7f64601385b0 addr v1:172.21.15.129:6789/0 2024-09-18T12:02:05.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460097270 con 0x7f64600a0750 2024-09-18T12:02:05.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64600a3170 con 0x7f64601385b0 2024-09-18T12:02:05.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460096cb0 con 0x7f64600a0380 2024-09-18T12:02:05.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient(hunting): _renew_subs 2024-09-18T12:02:05.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f646480e700 10 monclient(hunting): authenticate will time out at 1897.010591s 2024-09-18T12:02:05.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f6457fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f64600a0750 legacy=0x7f64600a0b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:05.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f645f7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f64600a0380 0x7f6460097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39938/0 (socket says 172.21.15.112:39938) 2024-09-18T12:02:05.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.168+0000 7f645f7fe700 1 -- 172.21.15.112:0/3385164423 learned_addr learned my addr 172.21.15.112:0/3385164423 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:02:05.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645f7fe700 1 -- 172.21.15.112:0/3385164423 >> v1:172.21.15.112:6790/0 conn(0x7f64600a0380 legacy=0x7f6460097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645ffff700 1 -- 172.21.15.112:0/3385164423 >> v1:172.21.15.129:6789/0 conn(0x7f64601385b0 legacy=0x7f646013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262492681 0 0) 0x7f6460096cb0 con 0x7f64600a0380 2024-09-18T12:02:05.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient(hunting): my global_id is 154169 2024-09-18T12:02:05.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6440003570 con 0x7f64600a0380 2024-09-18T12:02:05.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080735958 0 0) 0x7f64600a3170 con 0x7f64601385b0 2024-09-18T12:02:05.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient(hunting): my global_id is 144259 2024-09-18T12:02:05.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6460096cb0 con 0x7f64601385b0 2024-09-18T12:02:05.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 868821618 0 0) 0x7f6440003570 con 0x7f64600a0380 2024-09-18T12:02:05.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 >> v1:172.21.15.129:6789/0 conn(0x7f64601385b0 legacy=0x7f646013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 >> v1:172.21.15.112:6789/0 conn(0x7f64600a0750 legacy=0x7f64600a0b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:02:05.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: _finish_hunting 0 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 monclient: found mon.noname-b 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64600a1420 con 0x7f64600a0380 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: _finish_auth 0 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 5 monclient: authenticate success, global_id 154169 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6450001940 con 0x7f64600a0380 2024-09-18T12:02:05.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: dump: 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6450001bd0 con 0x7f64600a0380 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 10 monclient: get_monmap_and_config success 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 4 set_mon_vals no callback set 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 1 -- 172.21.15.112:0/3385164423 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6450004a90 con 0x7f64600a0380 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f645effd700 10 monclient: dump: 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.191 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 10 monclient: shutdown 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 1 -- 172.21.15.112:0/3385164423 >> v1:172.21.15.112:6790/0 conn(0x7f64600a0380 legacy=0x7f6460097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 1 -- 172.21.15.112:0/3385164423 shutdown_connections 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 1 -- 172.21.15.112:0/3385164423 >> 172.21.15.112:0/3385164423 conn(0x7f64600a5990 msgr2=0x7f64600a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 1 -- 172.21.15.112:0/3385164423 shutdown_connections 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.172+0000 7f646480e700 1 -- 172.21.15.112:0/3385164423 wait complete. 2024-09-18T12:02:05.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient: build_initial_monmap 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient: monmap: 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 librados: starting msgr at 2024-09-18T12:02:05.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 librados: starting objecter 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 Processor -- start 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 -- start start 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 librados: setting wanted keys 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 librados: calling monclient init 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient: init 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient: _reopen_session rank -1 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient(hunting): picked mon.noname-b con 0x7f64600a0380 addr v1:172.21.15.129:6789/0 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient(hunting): picked mon.noname-a con 0x7f64600a0750 addr v1:172.21.15.112:6789/0 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient(hunting): picked mon.noname-c con 0x7f64601385b0 addr v1:172.21.15.112:6790/0 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460138c20 con 0x7f64600a0750 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f6457fff700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f64600a0750 legacy=0x7f646011cbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:05.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646011ed90 con 0x7f64600a0380 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646011f070 con 0x7f64601385b0 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient(hunting): _renew_subs 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f646480e700 10 monclient(hunting): authenticate will time out at 1897.017996s 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f645ffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f64601385b0 0x7f6460149890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39940/0 (socket says 172.21.15.112:39940) 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.176+0000 7f645ffff700 1 -- 172.21.15.112:0/552467621 learned_addr learned my addr 172.21.15.112:0/552467621 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f645ffff700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f64601385b0 legacy=0x7f6460149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 client.?.objecter ms_handle_connect 0x7f64601385b0 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910021085 0 0) 0x7f646011f070 con 0x7f64601385b0 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): my global_id is 154172 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6444004f10 con 0x7f64601385b0 2024-09-18T12:02:05.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f645f7fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.129:6789/0 conn(0x7f64600a0380 legacy=0x7f646011c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 client.?.objecter ms_handle_connect 0x7f64600a0380 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399459295 0 0) 0x7f646011ed90 con 0x7f64600a0380 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): my global_id is 144262 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646011f070 con 0x7f64600a0380 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2769386795 0 0) 0x7f6444004f10 con 0x7f64601385b0 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f646011ed90 con 0x7f64601385b0 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6458002020 con 0x7f64601385b0 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:02:05.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): dump: 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3751907864 0 0) 0x7f646011ed90 con 0x7f64601385b0 2024-09-18T12:02:05.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.129:6789/0 conn(0x7f64600a0380 legacy=0x7f646011c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6789/0 conn(0x7f64600a0750 legacy=0x7f646011cbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient: _finish_hunting 0 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 monclient: found mon.c 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646009e080 con 0x7f64601385b0 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient: _finish_auth 0 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 5 monclient: authenticate success, global_id 154172 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 10 monclient: _renew_subs 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646011dc60 con 0x7f64601385b0 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 10 client.154172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:02:05.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 10 monclient: _renew_subs 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6460137ec0 con 0x7f64601385b0 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 1 librados: init done 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6458002630 con 0x7f64601385b0 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6458002980 con 0x7f64601385b0 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f645e7fc700 4 set_mon_vals no callback set 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 monclient: dump: 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.199 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:02:05.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f6458002ae0 con 0x7f64601385b0 2024-09-18T12:02:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6438006590 con 0x7f6438002840 2024-09-18T12:02:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(692..692 src has 1..692) v4 ==== 5806+0+0 (unknown 902576006 0 0) 0x7f64580442e0 con 0x7f64601385b0 2024-09-18T12:02:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 10 client.154172.objecter ms_dispatch 0x7f6460000fd0 osd_map(692..692 src has 1..692) v4 2024-09-18T12:02:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 3 client.154172.objecter handle_osd_map got epochs [692,692] > 0 2024-09-18T12:02:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 3 client.154172.objecter handle_osd_map decoding full epoch 692 2024-09-18T12:02:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f64577fe700 20 client.154172.objecter dump_active .. 0 homeless 2024-09-18T12:02:05.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.180+0000 7f6457fff700 10 monclient: get_auth_request con 0x7f64440381b0 auth_method 0 2024-09-18T12:02:05.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f645f7fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438002840 legacy=0x7f6438004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 client.154172.objecter ms_handle_connect 0x7f6438002840 2024-09-18T12:02:05.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 client.154172.objecter resend_mon_ops 2024-09-18T12:02:05.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068620372 0 0) 0x7f6438006590 con 0x7f6438002840 2024-09-18T12:02:05.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 monclient(hunting): my global_id is 154175 2024-09-18T12:02:05.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646011ed90 con 0x7f6438002840 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1434274685 0 0) 0x7f646011ed90 con 0x7f6438002840 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6438006590 con 0x7f6438002840 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6450001580 con 0x7f6438002840 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 monclient: dump: 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.206 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.207 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.207 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.207 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.207 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.207 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 client.154172.objecter ms_handle_connect 0x7f64440381b0 2024-09-18T12:02:05.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 471171360 0 0) 0x7f6438006590 con 0x7f6438002840 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f643800a140 con 0x7f6438002840 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6450001760 con 0x7f6438002840 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438002840 legacy=0x7f6438004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.184+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438002840 legacy=0x7f6438004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f646480e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6438009f10 con 0x7f6438006900 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f645f7fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438006900 legacy=0x7f6438006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:02:05.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f645f7fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438006900 legacy=0x7f6438006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f645f7fe700 1 --1- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438006900 0x7f6438006d80 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f645f7fe700 1 --1- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438006900 0x7f6438006d80 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438006900 legacy=0x7f6438006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6438006900 legacy=0x7f6438006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6438006590 con 0x7f6444043080 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f645f7fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6444043080 legacy=0x7f6444041a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 10 client.154172.objecter ms_handle_connect 0x7f6444043080 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 10 client.154172.objecter resend_mon_ops 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275023850 0 0) 0x7f6438006590 con 0x7f6444043080 2024-09-18T12:02:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 10 monclient(hunting): my global_id is 154178 2024-09-18T12:02:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646011ed90 con 0x7f6444043080 2024-09-18T12:02:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3347149387 0 0) 0x7f646011ed90 con 0x7f6444043080 2024-09-18T12:02:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6438006590 con 0x7f6444043080 2024-09-18T12:02:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6450005390 con 0x7f6444043080 2024-09-18T12:02:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:02:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 10 monclient: dump: 2024-09-18T12:02:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.200+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4213691960 0 0) 0x7f6438006590 con 0x7f6444043080 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6450001760 con 0x7f6444043080 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4160 (unknown 0 0 943899293) 0x7f64500075d0 con 0x7f6444043080 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f64577fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f64577fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6444043080 legacy=0x7f6444041a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f645ffff700 1 -- 172.21.15.112:0/552467621 reap_dead start 2024-09-18T12:02:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f64577fe700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f6444043080 legacy=0x7f6444041a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 78, 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:02:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:02:05.214 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:02:05.215 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:02:05.215 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.215 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:02:05.215 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:02:05.215 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:02:05.215 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:02:05.216 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:02:05.216 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:02:05.216 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:02:05.216 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:02:05.216 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:02:05.216 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:02:05.217 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:02:05.217 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.218 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:02:05.218 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:02:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:02:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:02:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:02:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:02:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:02:05.219 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:02:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:02:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:02:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:02:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:02:05.220 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:02:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:02:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:02:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:02:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:02:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:02:05.221 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:02:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:02:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:02:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:02:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:02:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:02:05.222 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:02:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:02:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:02:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:02:05.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:02:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:02:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:02:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:02:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.224 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:02:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:02:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:02:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.225 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:02:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:02:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:02:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:02:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:02:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:02:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:02:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:02:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:02:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.227 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:02:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:02:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:02:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:02:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:02:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:02:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:02:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:02:05.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:02:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:02:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:02:05.230 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:02:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:02:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:02:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:02:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.231 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:02:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:02:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:02:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:02:05.232 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:02:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:02:05.233 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:02:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:02:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:02:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:02:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.234 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.235 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:02:05.236 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 10 librados: watch_flush enter 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 10 librados: watch_flush exit 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 20 client.154172.objecter shutdown clearing up homeless session... 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 10 client.154172.objecter successfully canceled tick 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.129:6812/20806 conn(0x7f64440381b0 legacy=0x7f644403a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 10 monclient: shutdown 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 >> v1:172.21.15.112:6790/0 conn(0x7f64601385b0 legacy=0x7f6460149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 shutdown_connections 2024-09-18T12:02:05.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 >> 172.21.15.112:0/552467621 conn(0x7f64600a5990 msgr2=0x7f64600a4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:02:05.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 shutdown_connections 2024-09-18T12:02:05.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 1 -- 172.21.15.112:0/552467621 wait complete. 2024-09-18T12:02:05.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.204+0000 7f646480e700 1 librados: shutdown 2024-09-18T12:02:05.239 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:02:05.239 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:02:05.239 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:02:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.352+0000 7f8322a4e700 10 monclient: get_monmap_and_config 2024-09-18T12:02:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.352+0000 7f8322a4e700 10 monclient: build_initial_monmap 2024-09-18T12:02:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.352+0000 7f8322a4e700 10 monclient: monmap: 2024-09-18T12:02:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:02:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:02:05.356071+0000 2024-09-18T12:02:05.356 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:02:05.356071+0000 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 1 Processor -- start 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 1 -- start start 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 10 monclient: init 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 10 monclient: _reopen_session rank -1 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 10 monclient(hunting): picked mon.noname-a con 0x7f831c0a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T12:02:05.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 10 monclient(hunting): picked mon.noname-c con 0x7f831c06c280 addr v1:172.21.15.129:6789/0 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 10 monclient(hunting): picked mon.noname-b con 0x7f831c06c670 addr v1:172.21.15.112:6790/0 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c06e950 con 0x7f831c0a1ea0 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c0a2270 con 0x7f831c06c280 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c06d130 con 0x7f831c06c670 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 10 monclient(hunting): _renew_subs 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8322a4e700 10 monclient(hunting): authenticate will time out at 1897.196909s 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8321a4c700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f831c0a1ea0 legacy=0x7f831c06bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f832224d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f831c06c670 0x7f831c1388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39962/0 (socket says 172.21.15.112:39962) 2024-09-18T12:02:05.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f832224d700 1 -- 172.21.15.112:0/460440258 learned_addr learned my addr 172.21.15.112:0/460440258 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:02:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f832124b700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.129:6789/0 conn(0x7f831c06c280 legacy=0x7f831c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f832224d700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6790/0 conn(0x7f831c06c670 legacy=0x7f831c1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460354034 0 0) 0x7f831c0a2270 con 0x7f831c06c280 2024-09-18T12:02:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 10 monclient(hunting): my global_id is 144265 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8308003570 con 0x7f831c06c280 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989631938 0 0) 0x7f831c06d130 con 0x7f831c06c670 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 10 monclient(hunting): my global_id is 154181 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.356+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f831c0a2270 con 0x7f831c06c670 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950336007 0 0) 0x7f831c0a2270 con 0x7f831c06c670 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.129:6789/0 conn(0x7f831c06c280 legacy=0x7f831c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6789/0 conn(0x7f831c0a1ea0 legacy=0x7f831c06bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: _finish_hunting 0 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 monclient: found mon.noname-b 2024-09-18T12:02:05.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c09e370 con 0x7f831c06c670 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: _finish_auth 0 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8314001ce0 con 0x7f831c06c670 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: dump: 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.362 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 5 monclient: authenticate success, global_id 154181 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f832224d700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6790/0 conn(0x7f831c06c670 legacy=0x7f831c1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f832224d700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6790/0 conn(0x7f831c06c670 legacy=0x7f831c1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f832224d700 1 --1- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6790/0 conn(0x7f831c06c670 0x7f831c1388a0 :-1 s=OPENED pgs=9309 cs=1 l=1).handle_message read tag failed 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f832224d700 1 --1- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6790/0 conn(0x7f831c06c670 0x7f831c1388a0 :-1 s=OPENED pgs=9309 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: ms_handle_reset current mon v1:172.21.15.112:6790/0 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: _reopen_session rank -1 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6790/0 conn(0x7f831c06c670 legacy=0x7f831c1388a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:02:05.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient(hunting): picked mon.c con 0x7f8308004d10 addr v1:172.21.15.112:6790/0 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient(hunting): picked mon.a con 0x7f831c0a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient(hunting): picked mon.b con 0x7f831c06c280 addr v1:172.21.15.129:6789/0 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f831c0a2270 con 0x7f831c0a1ea0 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f831c06e950 con 0x7f831c06c280 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8308003570 con 0x7f8308004d10 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient(hunting): _renew_subs 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f832124b700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6789/0 conn(0x7f831c0a1ea0 legacy=0x7f8308009b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6790/0 2024-09-18T12:02:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8321a4c700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6790/0 conn(0x7f8308004d10 legacy=0x7f83080050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121152437 0 0) 0x7f8308003570 con 0x7f8308004d10 2024-09-18T12:02:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 --> v1:172.21.15.112:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f8308010e50 con 0x7f8308004d10 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f832224d700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.129:6789/0 conn(0x7f831c06c280 legacy=0x7f830800d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2097231887 0 0) 0x7f8308010e50 con 0x7f8308004d10 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.129:6789/0 conn(0x7f831c06c280 legacy=0x7f830800d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6789/0 conn(0x7f831c0a1ea0 legacy=0x7f8308009b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: _finish_hunting 0 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 monclient: found mon.c 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f831c06d130 con 0x7f8308004d10 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: _finish_auth 0 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:02:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8318004360 con 0x7f8308004d10 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: dump: 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.371 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 1 -- 172.21.15.112:0/460440258 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8318004530 con 0x7f8308004d10 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8320a4a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 10 monclient: get_monmap_and_config success 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 4 set_mon_vals no callback set 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 10 monclient: shutdown 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 1 -- 172.21.15.112:0/460440258 >> v1:172.21.15.112:6790/0 conn(0x7f8308004d10 legacy=0x7f83080050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 1 -- 172.21.15.112:0/460440258 shutdown_connections 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 1 -- 172.21.15.112:0/460440258 >> 172.21.15.112:0/460440258 conn(0x7f831c067da0 msgr2=0x7f831c0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 1 -- 172.21.15.112:0/460440258 shutdown_connections 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.360+0000 7f8322a4e700 1 -- 172.21.15.112:0/460440258 wait complete. 2024-09-18T12:02:05.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient: build_initial_monmap 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient: monmap: 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 librados: starting msgr at 2024-09-18T12:02:05.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 librados: starting objecter 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 Processor -- start 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 -- start start 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 librados: setting wanted keys 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 librados: calling monclient init 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient: init 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient: _reopen_session rank -1 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient(hunting): picked mon.noname-a con 0x7f8308004d10 addr v1:172.21.15.112:6789/0 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient(hunting): picked mon.noname-c con 0x7f831c06c280 addr v1:172.21.15.112:6790/0 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient(hunting): picked mon.noname-b con 0x7f831c0a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c06e1a0 con 0x7f8308004d10 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c0a0b00 con 0x7f831c0a1ea0 2024-09-18T12:02:05.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c13d9b0 con 0x7f831c06c280 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient(hunting): _renew_subs 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8322a4e700 10 monclient(hunting): authenticate will time out at 1897.205295s 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8321a4c700 1 -- >> v1:172.21.15.112:6789/0 conn(0x7f8308004d10 legacy=0x7f831c13d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.112:6789/0 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f832124b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f831c06c280 0x7f831c1464d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:39978/0 (socket says 172.21.15.112:39978) 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f832124b700 1 -- 172.21.15.112:0/3644722833 learned_addr learned my addr 172.21.15.112:0/3644722833 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f832124b700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.112:6790/0 conn(0x7f831c06c280 legacy=0x7f831c1464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 10 client.?.objecter ms_handle_connect 0x7f831c06c280 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378614609 0 0) 0x7f831c13d9b0 con 0x7f831c06c280 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 10 monclient(hunting): my global_id is 154184 2024-09-18T12:02:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8300004f10 con 0x7f831c06c280 2024-09-18T12:02:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 10 client.?.objecter ms_handle_connect 0x7f831c0a1ea0 2024-09-18T12:02:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f8311ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:02:05.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.364+0000 7f832224d700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6789/0 conn(0x7f831c0a1ea0 legacy=0x7f831c149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365164770 0 0) 0x7f831c0a0b00 con 0x7f831c0a1ea0 2024-09-18T12:02:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient(hunting): my global_id is 144268 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f831c13d9b0 con 0x7f831c0a1ea0 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3294999118 0 0) 0x7f8300004f10 con 0x7f831c06c280 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f831c0a0b00 con 0x7f831c06c280 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f830c002490 con 0x7f831c06c280 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient(hunting): dump: 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.379 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 717574084 0 0) 0x7f831c0a0b00 con 0x7f831c06c280 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6789/0 conn(0x7f831c0a1ea0 legacy=0x7f831c149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.112:6789/0 conn(0x7f8308004d10 legacy=0x7f831c13d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 monclient: found mon.c 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c13e7c0 con 0x7f831c06c280 2024-09-18T12:02:05.380 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient: _finish_auth 0 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 5 monclient: authenticate success, global_id 154184 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 10 monclient: _renew_subs 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831c122340 con 0x7f831c06c280 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 10 client.154184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 10 monclient: _renew_subs 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f831c097ef0 con 0x7f831c06c280 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 1 librados: init done 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f830c001c70 con 0x7f831c06c280 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:02:05.381 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8313fff700 4 set_mon_vals no callback set 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f830c002790 con 0x7f831c06c280 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 monclient: dump: 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:02:05.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f82fc006590 con 0x7f82fc002840 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f830c002970 con 0x7f831c06c280 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(692..692 src has 1..692) v4 ==== 5806+0+0 (unknown 902576006 0 0) 0x7f830c0441c0 con 0x7f831c06c280 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 10 client.154184.objecter ms_dispatch 0x7f831c000fd0 osd_map(692..692 src has 1..692) v4 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 3 client.154184.objecter handle_osd_map got epochs [692,692] > 0 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 3 client.154184.objecter handle_osd_map decoding full epoch 692 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f8311ffb700 20 client.154184.objecter dump_active .. 0 homeless 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.368+0000 7f832224d700 10 monclient: get_auth_request con 0x7f8300037e00 auth_method 0 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 client.154184.objecter ms_handle_connect 0x7f82fc002840 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 client.154184.objecter resend_mon_ops 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8321a4c700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6789/0 conn(0x7f82fc002840 legacy=0x7f82fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154658220 0 0) 0x7f82fc006590 con 0x7f82fc002840 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 monclient(hunting): my global_id is 144271 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f831c0a0b00 con 0x7f82fc002840 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2733949458 0 0) 0x7f831c0a0b00 con 0x7f82fc002840 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82fc006590 con 0x7f82fc002840 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f8318004330 con 0x7f82fc002840 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 monclient: dump: 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.386 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2057261421 0 0) 0x7f82fc006590 con 0x7f82fc002840 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82fc00a140 con 0x7f82fc002840 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 client.154184.objecter ms_handle_connect 0x7f8300037e00 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f8318005910 con 0x7f82fc002840 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6789/0 conn(0x7f82fc002840 legacy=0x7f82fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.372+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6789/0 conn(0x7f82fc002840 legacy=0x7f82fc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:02:05.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8322a4e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:02:05.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f82fc007550 con 0x7f82fc006900 2024-09-18T12:02:05.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8321a4c700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6789/0 conn(0x7f82fc006900 legacy=0x7f82fc006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:02:05.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 10 client.154184.objecter ms_handle_connect 0x7f82fc006900 2024-09-18T12:02:05.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 10 client.154184.objecter resend_mon_ops 2024-09-18T12:02:05.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765922789 0 0) 0x7f82fc007550 con 0x7f82fc006900 2024-09-18T12:02:05.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 10 monclient(hunting): my global_id is 144274 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82fc006590 con 0x7f82fc006900 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3473080756 0 0) 0x7f82fc006590 con 0x7f82fc006900 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82fc007550 con 0x7f82fc006900 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f83180046e0 con 0x7f82fc006900 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.384+0000 7f8311ffb700 10 monclient: dump: 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2973451452 0 0) 0x7f82fc007550 con 0x7f82fc006900 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f82fc0087b0 con 0x7f82fc006900 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3992 (unknown 0 0 699434789) 0x7f8318004940 con 0x7f82fc006900 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8311ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8311ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6789/0 conn(0x7f82fc006900 legacy=0x7f82fc006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8311ffb700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6789/0 conn(0x7f82fc006900 legacy=0x7f82fc006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 78, 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:02:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:02:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:02:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:02:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:02:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:02:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 3 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:02:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:02:05.403 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:02:05.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 10 librados: watch_flush enter 2024-09-18T12:02:05.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 10 librados: watch_flush exit 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 20 client.154184.objecter shutdown clearing up homeless session... 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 10 client.154184.objecter successfully canceled tick 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.129:6812/20806 conn(0x7f8300037e00 legacy=0x7f830003a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 10 monclient: shutdown 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 >> v1:172.21.15.112:6790/0 conn(0x7f831c06c280 legacy=0x7f831c1464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f832224d700 1 -- 172.21.15.112:0/3644722833 reap_dead start 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f832224d700 1 -- 172.21.15.112:0/3644722833 reap_dead start 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 shutdown_connections 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 >> 172.21.15.112:0/3644722833 conn(0x7f831c067da0 msgr2=0x7f831c09be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 shutdown_connections 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 1 -- 172.21.15.112:0/3644722833 wait complete. 2024-09-18T12:02:05.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:02:05.388+0000 7f8322a4e700 1 librados: shutdown 2024-09-18T12:02:05.406 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:02:05.406 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-18T12:02:11.918 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7864 ms) 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13096 ms) 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7926 ms) 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13021 ms) 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7982 ms) 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8015 ms) 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-18T12:02:11.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (22186 ms) 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first is 1726660830 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,1726660835,1726660836,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,1726660835,1726660836,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,1726660835,1726660836,0 2024-09-18T12:02:11.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,1726660835,1726660836,1726660838,1726660839,0 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,1726660835,1726660836,1726660838,1726660839,0 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,1726660835,1726660836,1726660838,1726660841,0 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,1726660835,1726660836,1726660838,1726660841,0 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660830,1726660832,1726660833,1726660835,1726660836,1726660838,1726660841,0 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1726660833,1726660835,1726660836,1726660838,1726660841,1726660843,1726660844,0 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first now 1726660833, trimmed 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (23857 ms) 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: foo0 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13959 ms) 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18292 ms) 2024-09-18T12:02:11.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22748 ms) 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5047 ms) 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3155 ms) 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (17174 ms) 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3028 ms) 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (279033 ms total) 2024-09-18T12:02:11.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2024-09-18T12:02:11.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-18T12:02:11.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1007964 ms total) 2024-09-18T12:02:11.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-18T12:02:11.925 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.925 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24083 2024-09-18T12:02:11.925 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24083 2024-09-18T12:02:11.925 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23815 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23815 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23792 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23792 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23862 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23862 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24124 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24124 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24327 2024-09-18T12:02:11.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24327 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24014 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24014 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23768 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23768 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24402 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24402 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24219 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24219 2024-09-18T12:02:11.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24050 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24050 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24377 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24377 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23800 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23800 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24273 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24273 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23784 2024-09-18T12:02:11.928 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23784 2024-09-18T12:02:11.929 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.929 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24176 2024-09-18T12:02:11.929 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24176 2024-09-18T12:02:11.929 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.929 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=23933 2024-09-18T12:02:11.929 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 23933 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24204 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24204 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=24028 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 24028 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ exit 0 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ cleanup 2024-09-18T12:02:11.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ pkill -P 23754 2024-09-18T12:02:11.947 INFO:tasks.workunit.client.0.smithi129.stderr:+ true 2024-09-18T12:02:11.948 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T12:02:11.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T12:02:11.960 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-09-18T12:02:11.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-18T12:02:12.273 DEBUG:teuthology.parallel:result is None 2024-09-18T12:02:12.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:02:12.289 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:02:12.289 DEBUG:teuthology.orchestra.run.smithi129:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T12:02:12.311 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:02:12.311 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-18T12:02:12.319 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-18T12:03:35.407 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:03:35.407 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T12:03:35.407 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:03:35.408 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:03:35.411 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:03:35.411 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:03:35.411 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:03:35.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient: get_monmap_and_config 2024-09-18T12:03:35.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient: build_initial_monmap 2024-09-18T12:03:35.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient: monmap: 2024-09-18T12:03:35.512 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:35.513088+0000 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:35.513088+0000 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 1 Processor -- start 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 1 -- start start 2024-09-18T12:03:35.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient: init 2024-09-18T12:03:35.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:35.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:03:35.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient(hunting): picked mon.noname-a con 0x7f609009ae70 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:35.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient(hunting): picked mon.noname-b con 0x7f6090132860 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:35.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient(hunting): picked mon.noname-c con 0x7f6090132c60 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:35.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609006e750 con 0x7f609009ae70 2024-09-18T12:03:35.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609009b240 con 0x7f6090132c60 2024-09-18T12:03:35.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609009b660 con 0x7f6090132860 2024-09-18T12:03:35.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:35.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f6095ea2700 10 monclient(hunting): authenticate will time out at 1987.354010s 2024-09-18T12:03:35.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f608f7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f609009ae70 0x7f6090132150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51842/0 (socket says 172.21.15.112:51842) 2024-09-18T12:03:35.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.509+0000 7f608f7fe700 1 -- 172.21.15.112:0/974339575 learned_addr learned my addr 172.21.15.112:0/974339575 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608f7fe700 1 -- 172.21.15.112:0/974339575 >> v1:172.21.15.112:6789/0 conn(0x7f609009ae70 legacy=0x7f6090132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608effd700 1 -- 172.21.15.112:0/974339575 >> v1:172.21.15.112:6790/0 conn(0x7f6090132860 legacy=0x7f6090137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151188006 0 0) 0x7f609009b660 con 0x7f6090132860 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient(hunting): my global_id is 154187 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6078003570 con 0x7f6090132860 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3509131517 0 0) 0x7f6078003570 con 0x7f6090132860 2024-09-18T12:03:35.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 >> v1:172.21.15.129:6789/0 conn(0x7f6090132c60 legacy=0x7f609013abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 >> v1:172.21.15.112:6789/0 conn(0x7f609009ae70 legacy=0x7f6090132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 monclient: found mon.noname-b 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609009b8a0 con 0x7f6090132860 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: _finish_auth 0 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 5 monclient: authenticate success, global_id 154187 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6080001a50 con 0x7f6090132860 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: dump: 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:35.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6080001c70 con 0x7f6090132860 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 1 -- 172.21.15.112:0/974339575 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f60800020a0 con 0x7f6090132860 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 4 set_mon_vals no callback set 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f608e7fc700 10 monclient: dump: 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.521 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 10 monclient: shutdown 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 -- 172.21.15.112:0/974339575 >> v1:172.21.15.112:6790/0 conn(0x7f6090132860 legacy=0x7f6090137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 -- 172.21.15.112:0/974339575 shutdown_connections 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 -- 172.21.15.112:0/974339575 >> 172.21.15.112:0/974339575 conn(0x7f60900680d0 msgr2=0x7f60900a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 -- 172.21.15.112:0/974339575 shutdown_connections 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 -- 172.21.15.112:0/974339575 wait complete. 2024-09-18T12:03:35.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 10 monclient: build_initial_monmap 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 10 monclient: monmap: 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:35.523 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 librados: starting msgr at 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 librados: starting objecter 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 Processor -- start 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 -- start start 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 librados: setting wanted keys 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 1 librados: calling monclient init 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.513+0000 7f6095ea2700 10 monclient: init 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:35.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 10 monclient(hunting): picked mon.noname-a con 0x7f609009ae70 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 10 monclient(hunting): picked mon.noname-b con 0x7f6090132860 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 10 monclient(hunting): picked mon.noname-c con 0x7f6090132c60 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900a0ef0 con 0x7f609009ae70 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6090132230 con 0x7f6090132860 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609011e740 con 0x7f6090132c60 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f6095ea2700 10 monclient(hunting): authenticate will time out at 1987.360950s 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f608f7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f609009ae70 0x7f6090141620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51850/0 (socket says 172.21.15.112:51850) 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f608f7fe700 1 -- 172.21.15.112:0/2016902740 learned_addr learned my addr 172.21.15.112:0/2016902740 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f608ffff700 1 -- 172.21.15.112:0/2016902740 >> v1:172.21.15.112:6790/0 conn(0x7f6090132c60 legacy=0x7f6090149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f60777fe700 10 client.?.objecter ms_handle_connect 0x7f6090132c60 2024-09-18T12:03:35.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f60777fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:35.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f608f7fe700 1 -- 172.21.15.112:0/2016902740 >> v1:172.21.15.112:6789/0 conn(0x7f609009ae70 legacy=0x7f6090141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.517+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:03:35.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 client.?.objecter ms_handle_connect 0x7f609009ae70 2024-09-18T12:03:35.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:35.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758508530 0 0) 0x7f609011e740 con 0x7f6090132c60 2024-09-18T12:03:35.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 monclient(hunting): my global_id is 154190 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607c004f10 con 0x7f6090132c60 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227695252 0 0) 0x7f607c004f10 con 0x7f6090132c60 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f609011e740 con 0x7f6090132c60 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6084002bb0 con 0x7f6090132c60 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 monclient(hunting): dump: 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.529 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 client.?.objecter ms_handle_connect 0x7f6090132860 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f608effd700 1 -- 172.21.15.112:0/2016902740 >> v1:172.21.15.129:6789/0 conn(0x7f6090132860 legacy=0x7f60901312b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.521+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597541146 0 0) 0x7f6090132230 con 0x7f6090132860 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient(hunting): my global_id is 144277 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607c004f10 con 0x7f6090132860 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 924256006 0 0) 0x7f609011e740 con 0x7f6090132c60 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 >> v1:172.21.15.129:6789/0 conn(0x7f6090132860 legacy=0x7f60901312b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 >> v1:172.21.15.112:6789/0 conn(0x7f609009ae70 legacy=0x7f6090141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient: _finish_hunting 0 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 monclient: found mon.c 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609011e3d0 con 0x7f6090132c60 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient: _finish_auth 0 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 5 monclient: authenticate success, global_id 154190 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 10 monclient: _renew_subs 2024-09-18T12:03:35.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609011c8a0 con 0x7f6090132c60 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 10 client.154190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 10 monclient: _renew_subs 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60901319c0 con 0x7f6090132c60 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 1 librados: init done 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6084001280 con 0x7f6090132c60 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6084001590 con 0x7f6090132c60 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 monclient: dump: 2024-09-18T12:03:35.532 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f608dffb700 4 set_mon_vals no callback set 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f60840017e0 con 0x7f6090132c60 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f6084043b30 con 0x7f6090132c60 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 10 client.154190.objecter ms_dispatch 0x7f6090000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 3 client.154190.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 3 client.154190.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f60777fe700 20 client.154190.objecter dump_active .. 0 homeless 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f608f7fe700 10 monclient: get_auth_request con 0x7f607c037f80 auth_method 0 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.525+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60700050b0 con 0x7f6090132c60 2024-09-18T12:03:35.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.529+0000 7f60777fe700 10 client.154190.objecter ms_handle_connect 0x7f607c037f80 2024-09-18T12:03:35.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.533+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f6084043e60 con 0x7f6090132c60 2024-09-18T12:03:35.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.533+0000 7f60777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:35.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.533+0000 7f60777fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:35.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:03:35.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:03:35.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:35.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f60700058e0 con 0x7f6090132c60 2024-09-18T12:03:35.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f60777fe700 1 -- 172.21.15.112:0/2016902740 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1361936002) 0x7f60840145d0 con 0x7f6090132c60 2024-09-18T12:03:35.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f60777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:03:35.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f60777fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:03:35.769 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 10 librados: watch_flush enter 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 10 librados: watch_flush exit 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 20 client.154190.objecter shutdown clearing up homeless session... 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 10 client.154190.objecter successfully canceled tick 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 >> v1:172.21.15.129:6812/20806 conn(0x7f607c037f80 legacy=0x7f607c03a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 10 monclient: shutdown 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 >> v1:172.21.15.112:6790/0 conn(0x7f6090132c60 legacy=0x7f6090149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 shutdown_connections 2024-09-18T12:03:35.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.765+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 >> 172.21.15.112:0/2016902740 conn(0x7f60900680d0 msgr2=0x7f609006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:35.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.769+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 shutdown_connections 2024-09-18T12:03:35.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.769+0000 7f6095ea2700 1 -- 172.21.15.112:0/2016902740 wait complete. 2024-09-18T12:03:35.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.769+0000 7f6095ea2700 1 librados: shutdown 2024-09-18T12:03:35.782 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:03:35.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:03:35.783 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:03:35.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.877+0000 7f08c7436700 10 monclient: get_monmap_and_config 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.877+0000 7f08c7436700 10 monclient: build_initial_monmap 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.877+0000 7f08c7436700 10 monclient: monmap: 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:35.883718+0000 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:35.883718+0000 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:35.883 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.877+0000 7f08c7436700 1 Processor -- start 2024-09-18T12:03:35.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.877+0000 7f08c7436700 1 -- start start 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.877+0000 7f08c7436700 10 monclient: init 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 10 monclient(hunting): picked mon.noname-a con 0x7f08c00a2040 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 10 monclient(hunting): picked mon.noname-c con 0x7f08c012edc0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 10 monclient(hunting): picked mon.noname-b con 0x7f08c012f8a0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c00981a0 con 0x7f08c00a2040 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c006c120 con 0x7f08c012edc0 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c006eef0 con 0x7f08c012f8a0 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 10 monclient(hunting): authenticate will time out at 1987.724567s 2024-09-18T12:03:35.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c6434700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f08c00a2040 0x7f08c0097a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51860/0 (socket says 172.21.15.112:51860) 2024-09-18T12:03:35.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c6434700 1 -- 172.21.15.112:0/3120904320 learned_addr learned my addr 172.21.15.112:0/3120904320 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:35.886 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c6c35700 1 --1- 172.21.15.112:0/3120904320 >> v1:172.21.15.112:6790/0 conn(0x7f08c012f8a0 0x7f08c0137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35546/0 (socket says 172.21.15.112:35546) 2024-09-18T12:03:35.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c6434700 1 -- 172.21.15.112:0/3120904320 >> v1:172.21.15.112:6789/0 conn(0x7f08c00a2040 legacy=0x7f08c0097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c6c35700 1 -- 172.21.15.112:0/3120904320 >> v1:172.21.15.112:6790/0 conn(0x7f08c012f8a0 legacy=0x7f08c0137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5c33700 1 -- 172.21.15.112:0/3120904320 >> v1:172.21.15.129:6789/0 conn(0x7f08c012edc0 legacy=0x7f08c012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513493685 0 0) 0x7f08c006eef0 con 0x7f08c012f8a0 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): my global_id is 154193 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a8003570 con 0x7f08c012f8a0 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559402489 0 0) 0x7f08c00981a0 con 0x7f08c00a2040 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): my global_id is 164097 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c006eef0 con 0x7f08c00a2040 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952831306 0 0) 0x7f08c006c120 con 0x7f08c012edc0 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient(hunting): my global_id is 144280 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c00981a0 con 0x7f08c012edc0 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 906414963 0 0) 0x7f08c006eef0 con 0x7f08c00a2040 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 >> v1:172.21.15.112:6790/0 conn(0x7f08c012f8a0 legacy=0x7f08c0137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 >> v1:172.21.15.129:6789/0 conn(0x7f08c012edc0 legacy=0x7f08c012f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient: _finish_hunting 0 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 monclient: found mon.noname-a 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0096dd0 con 0x7f08c00a2040 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient: _finish_auth 0 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 5 monclient: authenticate success, global_id 164097 2024-09-18T12:03:35.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08b4001920 con 0x7f08c00a2040 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c5432700 10 monclient: dump: 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.891 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.881+0000 7f08c7436700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08b4001bb0 con 0x7f08c00a2040 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c5432700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 4 set_mon_vals no callback set 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c5432700 1 -- 172.21.15.112:0/3120904320 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08b4001da0 con 0x7f08c00a2040 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c5432700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c5432700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c5432700 10 monclient: dump: 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.892 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient: shutdown 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- 172.21.15.112:0/3120904320 >> v1:172.21.15.112:6789/0 conn(0x7f08c00a2040 legacy=0x7f08c0097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- 172.21.15.112:0/3120904320 shutdown_connections 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- 172.21.15.112:0/3120904320 >> 172.21.15.112:0/3120904320 conn(0x7f08c0067da0 msgr2=0x7f08c00a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- 172.21.15.112:0/3120904320 shutdown_connections 2024-09-18T12:03:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- 172.21.15.112:0/3120904320 wait complete. 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient: build_initial_monmap 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient: monmap: 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 librados: starting msgr at 2024-09-18T12:03:35.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 librados: starting objecter 2024-09-18T12:03:35.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 Processor -- start 2024-09-18T12:03:35.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- start start 2024-09-18T12:03:35.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 librados: setting wanted keys 2024-09-18T12:03:35.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 librados: calling monclient init 2024-09-18T12:03:35.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient: init 2024-09-18T12:03:35.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient(hunting): picked mon.noname-c con 0x7f08c00a2040 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient(hunting): picked mon.noname-b con 0x7f08c012edc0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient(hunting): picked mon.noname-a con 0x7f08c012f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c006db40 con 0x7f08c012f8a0 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c011b300 con 0x7f08c012edc0 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c0132890 con 0x7f08c00a2040 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:35.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c7436700 10 monclient(hunting): authenticate will time out at 1987.730736s 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c6434700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f08c00a2040 0x7f08c012df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35560/0 (socket says 172.21.15.112:35560) 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.885+0000 7f08c6434700 1 -- 172.21.15.112:0/1593992164 learned_addr learned my addr 172.21.15.112:0/1593992164 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c6434700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6790/0 conn(0x7f08c00a2040 legacy=0x7f08c012df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 client.?.objecter ms_handle_connect 0x7f08c00a2040 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 client.?.objecter ms_handle_connect 0x7f08c012f8a0 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c6c35700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6789/0 conn(0x7f08c012f8a0 legacy=0x7f08c0145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 client.?.objecter ms_handle_connect 0x7f08c012edc0 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:35.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c5c33700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.129:6789/0 conn(0x7f08c012edc0 legacy=0x7f08c0142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729608565 0 0) 0x7f08c0132890 con 0x7f08c00a2040 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): my global_id is 154196 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ac004f10 con 0x7f08c00a2040 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959159180 0 0) 0x7f08c006db40 con 0x7f08c012f8a0 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): my global_id is 164100 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c0132890 con 0x7f08c012f8a0 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738345361 0 0) 0x7f08c011b300 con 0x7f08c012edc0 2024-09-18T12:03:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): my global_id is 144283 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c006db40 con 0x7f08c012edc0 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374557995 0 0) 0x7f08ac004f10 con 0x7f08c00a2040 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08c011b300 con 0x7f08c00a2040 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08b4004830 con 0x7f08c00a2040 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): dump: 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.905 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3455595809 0 0) 0x7f08c0132890 con 0x7f08c012f8a0 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08ac004f10 con 0x7f08c012f8a0 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08b80015e0 con 0x7f08c012f8a0 2024-09-18T12:03:35.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient(hunting): dump: 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.907 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1200539220 0 0) 0x7f08ac004f10 con 0x7f08c012f8a0 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6790/0 conn(0x7f08c00a2040 legacy=0x7f08c012df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.129:6789/0 conn(0x7f08c012edc0 legacy=0x7f08c0142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 monclient: found mon.a 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c01194f0 con 0x7f08c012f8a0 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient: _finish_auth 0 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:35.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 5 monclient: authenticate success, global_id 164100 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 10 monclient: _renew_subs 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c011b020 con 0x7f08c012f8a0 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 10 client.164100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 10 monclient: _renew_subs 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c012e7b0 con 0x7f08c012f8a0 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 1 librados: init done 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08b8004ae0 con 0x7f08c012f8a0 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08b8004df0 con 0x7f08c012f8a0 2024-09-18T12:03:35.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 monclient: dump: 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c4c31700 4 set_mon_vals no callback set 2024-09-18T12:03:35.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f08b800f6f0 con 0x7f08c012f8a0 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f08b8043ac0 con 0x7f08c012f8a0 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 10 client.164100.objecter ms_dispatch 0x7f08c0000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 3 client.164100.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 3 client.164100.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f08a0006590 con 0x7f08a0002840 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.889+0000 7f08be7fc700 20 client.164100.objecter dump_active .. 0 homeless 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08c6434700 10 monclient: get_auth_request con 0x7f08ac038410 auth_method 0 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08c5c33700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6789/0 conn(0x7f08a0002840 legacy=0x7f08a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 client.164100.objecter ms_handle_connect 0x7f08a0002840 2024-09-18T12:03:35.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 client.164100.objecter resend_mon_ops 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554134063 0 0) 0x7f08a0006590 con 0x7f08a0002840 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 monclient(hunting): my global_id is 164103 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ac004f10 con 0x7f08a0002840 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 client.164100.objecter ms_handle_connect 0x7f08ac038410 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 409062632 0 0) 0x7f08ac004f10 con 0x7f08a0002840 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a0006590 con 0x7f08a0002840 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08b0001ee0 con 0x7f08a0002840 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 monclient: dump: 2024-09-18T12:03:35.912 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 354580760 0 0) 0x7f08a0006590 con 0x7f08a0002840 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08a000a140 con 0x7f08a0002840 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f08b0005b90 con 0x7f08a0002840 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:35.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6789/0 conn(0x7f08a0002840 legacy=0x7f08a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.893+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6789/0 conn(0x7f08a0002840 legacy=0x7f08a0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:35.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.905+0000 7f08c7436700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:03:35.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.905+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f08a0007520 con 0x7f08a00068d0 2024-09-18T12:03:35.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.905+0000 7f08c5c33700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6789/0 conn(0x7f08a00068d0 legacy=0x7f08a0006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:35.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.905+0000 7f08be7fc700 10 client.164100.objecter ms_handle_connect 0x7f08a00068d0 2024-09-18T12:03:35.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.905+0000 7f08be7fc700 10 client.164100.objecter resend_mon_ops 2024-09-18T12:03:35.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541664436 0 0) 0x7f08a0007520 con 0x7f08a00068d0 2024-09-18T12:03:35.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 10 monclient(hunting): my global_id is 164106 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a0006590 con 0x7f08a00068d0 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1075449236 0 0) 0x7f08a0006590 con 0x7f08a00068d0 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a0007520 con 0x7f08a00068d0 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08b0002750 con 0x7f08a00068d0 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 10 monclient: dump: 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:35.916 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 632822860 0 0) 0x7f08a0007520 con 0x7f08a00068d0 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 --> v1:172.21.15.112:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f08a0008760 con 0x7f08a00068d0 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 <== mon.0 v1:172.21.15.112:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3833 (unknown 0 0 3030264477) 0x7f08b0002920 con 0x7f08a00068d0 2024-09-18T12:03:35.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:03:35.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:03:35.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6789/0 conn(0x7f08a00068d0 legacy=0x7f08a0006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08be7fc700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6789/0 conn(0x7f08a00068d0 legacy=0x7f08a0006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:35.918 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:03:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:03:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "leader", 2024-09-18T12:03:35.918 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 82, 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:35.919 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:35.920 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:03:35.921 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:03:35.922 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:35.923 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:35.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:35.925 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:35.926 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:03:35.927 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:35.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:35.929 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T12:03:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:35.930 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:03:35.930 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:35.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 10 librados: watch_flush enter 2024-09-18T12:03:35.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 10 librados: watch_flush exit 2024-09-18T12:03:35.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 20 client.164100.objecter shutdown clearing up homeless session... 2024-09-18T12:03:35.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 10 client.164100.objecter successfully canceled tick 2024-09-18T12:03:35.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.129:6812/20806 conn(0x7f08ac038410 legacy=0x7f08ac03a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 10 monclient: shutdown 2024-09-18T12:03:35.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c6c35700 1 -- 172.21.15.112:0/1593992164 reap_dead start 2024-09-18T12:03:35.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:35.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 >> v1:172.21.15.112:6789/0 conn(0x7f08c012f8a0 legacy=0x7f08c0145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:35.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 shutdown_connections 2024-09-18T12:03:35.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 >> 172.21.15.112:0/1593992164 conn(0x7f08c0067da0 msgr2=0x7f08c0098360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:35.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 shutdown_connections 2024-09-18T12:03:35.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 1 -- 172.21.15.112:0/1593992164 wait complete. 2024-09-18T12:03:35.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:35.909+0000 7f08c7436700 1 librados: shutdown 2024-09-18T12:03:35.933 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:03:35.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:03:35.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:03:36.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.037+0000 7fec5944d700 10 monclient: get_monmap_and_config 2024-09-18T12:03:36.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.037+0000 7fec5944d700 10 monclient: build_initial_monmap 2024-09-18T12:03:36.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.037+0000 7fec5944d700 10 monclient: monmap: 2024-09-18T12:03:36.043 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:36.043 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.043 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:36.043969+0000 2024-09-18T12:03:36.043 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:36.043969+0000 2024-09-18T12:03:36.043 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:36.044 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.044 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:36.044 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:36.044 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:36.044 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 1 Processor -- start 2024-09-18T12:03:36.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 1 -- start start 2024-09-18T12:03:36.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 10 monclient: init 2024-09-18T12:03:36.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:36.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:36.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 10 monclient(hunting): picked mon.noname-a con 0x7fec540a1ea0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:36.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 10 monclient(hunting): picked mon.noname-c con 0x7fec5406c280 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 10 monclient(hunting): picked mon.noname-b con 0x7fec5406c670 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5406e950 con 0x7fec540a1ea0 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec540a2270 con 0x7fec5406c280 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5406d130 con 0x7fec5406c670 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec5944d700 10 monclient(hunting): authenticate will time out at 1987.884920s 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec53fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fec540a1ea0 0x7fec5406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51902/0 (socket says 172.21.15.112:51902) 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec58c4c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fec5406c670 0x7fec541388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35564/0 (socket says 172.21.15.112:35564) 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec53fff700 1 -- 172.21.15.112:0/472236063 learned_addr learned my addr 172.21.15.112:0/472236063 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec4b7fe700 1 -- 172.21.15.112:0/472236063 >> v1:172.21.15.129:6789/0 conn(0x7fec5406c280 legacy=0x7fec54135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec53fff700 1 -- 172.21.15.112:0/472236063 >> v1:172.21.15.112:6789/0 conn(0x7fec540a1ea0 legacy=0x7fec5406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec58c4c700 1 -- 172.21.15.112:0/472236063 >> v1:172.21.15.112:6790/0 conn(0x7fec5406c670 legacy=0x7fec541388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054084099 0 0) 0x7fec540a2270 con 0x7fec5406c280 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec537fe700 10 monclient(hunting): my global_id is 144286 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.041+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec40003570 con 0x7fec5406c280 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885605338 0 0) 0x7fec5406e950 con 0x7fec540a1ea0 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient(hunting): my global_id is 164109 2024-09-18T12:03:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec540a2270 con 0x7fec540a1ea0 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721385234 0 0) 0x7fec5406d130 con 0x7fec5406c670 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient(hunting): my global_id is 154199 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec5406e950 con 0x7fec5406c670 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2831981745 0 0) 0x7fec540a2270 con 0x7fec540a1ea0 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 >> v1:172.21.15.112:6790/0 conn(0x7fec5406c670 legacy=0x7fec541388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 >> v1:172.21.15.129:6789/0 conn(0x7fec5406c280 legacy=0x7fec54135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: _finish_hunting 0 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 monclient: found mon.noname-a 2024-09-18T12:03:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5409e370 con 0x7fec540a1ea0 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: _finish_auth 0 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fec44001920 con 0x7fec540a1ea0 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 5 monclient: authenticate success, global_id 164109 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: dump: 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.053 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec44001bb0 con 0x7fec540a1ea0 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 1 -- 172.21.15.112:0/472236063 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fec44001da0 con 0x7fec540a1ea0 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:36.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec537fe700 10 monclient: dump: 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 4 set_mon_vals no callback set 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient: shutdown 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:36.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- 172.21.15.112:0/472236063 >> v1:172.21.15.112:6789/0 conn(0x7fec540a1ea0 legacy=0x7fec5406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- 172.21.15.112:0/472236063 shutdown_connections 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- 172.21.15.112:0/472236063 >> 172.21.15.112:0/472236063 conn(0x7fec54067da0 msgr2=0x7fec540a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- 172.21.15.112:0/472236063 shutdown_connections 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- 172.21.15.112:0/472236063 wait complete. 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient: build_initial_monmap 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient: monmap: 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:36.056 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:36.057 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:36.057 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.057 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:36.057 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:36.057 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:36.057 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 librados: starting msgr at 2024-09-18T12:03:36.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 librados: starting objecter 2024-09-18T12:03:36.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 Processor -- start 2024-09-18T12:03:36.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- start start 2024-09-18T12:03:36.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 librados: setting wanted keys 2024-09-18T12:03:36.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 librados: calling monclient init 2024-09-18T12:03:36.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient: init 2024-09-18T12:03:36.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient(hunting): picked mon.noname-a con 0x7fec5406c280 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient(hunting): picked mon.noname-c con 0x7fec5406c670 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient(hunting): picked mon.noname-b con 0x7fec540a1ea0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5406e1a0 con 0x7fec5406c280 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5411f470 con 0x7fec540a1ea0 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec54120760 con 0x7fec5406c670 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec5944d700 10 monclient(hunting): authenticate will time out at 1987.891346s 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec4b7fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fec5406c670 0x7fec54131500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35574/0 (socket says 172.21.15.112:35574) 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.045+0000 7fec4b7fe700 1 -- 172.21.15.112:0/4250247642 learned_addr learned my addr 172.21.15.112:0/4250247642 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec4b7fe700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6790/0 conn(0x7fec5406c670 legacy=0x7fec54131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 client.?.objecter ms_handle_connect 0x7fec5406c670 2024-09-18T12:03:36.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 client.?.objecter ms_handle_connect 0x7fec5406c280 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec53fff700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6789/0 conn(0x7fec5406c280 legacy=0x7fec541412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 client.?.objecter ms_handle_connect 0x7fec540a1ea0 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec58c4c700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.129:6789/0 conn(0x7fec540a1ea0 legacy=0x7fec54149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237384405 0 0) 0x7fec54120760 con 0x7fec5406c670 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): my global_id is 154202 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec34004f10 con 0x7fec5406c670 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043796405 0 0) 0x7fec5406e1a0 con 0x7fec5406c280 2024-09-18T12:03:36.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): my global_id is 164112 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec54120760 con 0x7fec5406c280 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214786079 0 0) 0x7fec5411f470 con 0x7fec540a1ea0 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): my global_id is 144289 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec5406e1a0 con 0x7fec540a1ea0 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3830468485 0 0) 0x7fec34004f10 con 0x7fec5406c670 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec5411f470 con 0x7fec5406c670 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fec3c001f00 con 0x7fec5406c670 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): dump: 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:36.068 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 543920267 0 0) 0x7fec54120760 con 0x7fec5406c280 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec34004f10 con 0x7fec5406c280 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fec44004830 con 0x7fec5406c280 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): dump: 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.069 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 713640458 0 0) 0x7fec5406e1a0 con 0x7fec540a1ea0 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec54120760 con 0x7fec540a1ea0 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fec4c004bb0 con 0x7fec540a1ea0 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient(hunting): dump: 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.070 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2135454040 0 0) 0x7fec34004f10 con 0x7fec5406c280 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6790/0 conn(0x7fec5406c670 legacy=0x7fec54131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.129:6789/0 conn(0x7fec540a1ea0 legacy=0x7fec54149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient: _finish_hunting 0 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 monclient: found mon.a 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:36.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec54121650 con 0x7fec5406c280 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient: _finish_auth 0 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 5 monclient: authenticate success, global_id 164112 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 10 monclient: _renew_subs 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec54121930 con 0x7fec5406c280 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 10 client.164112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 10 monclient: _renew_subs 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec541419e0 con 0x7fec5406c280 2024-09-18T12:03:36.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec5944d700 1 librados: init done 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec44004ee0 con 0x7fec5406c280 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fec440051f0 con 0x7fec5406c280 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec52ffd700 4 set_mon_vals no callback set 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.049+0000 7fec50ff9700 10 monclient: dump: 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.073 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec5944d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fec38006590 con 0x7fec38002840 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fec4400fa70 con 0x7fec5406c280 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fec44043c60 con 0x7fec5406c280 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 client.164112.objecter ms_dispatch 0x7fec54000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 3 client.164112.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 3 client.164112.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 20 client.164112.objecter dump_active .. 0 homeless 2024-09-18T12:03:36.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec58c4c700 10 monclient: get_auth_request con 0x7fec34038580 auth_method 0 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 client.164112.objecter ms_handle_connect 0x7fec38002840 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 client.164112.objecter resend_mon_ops 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec4b7fe700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6790/0 conn(0x7fec38002840 legacy=0x7fec38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846484298 0 0) 0x7fec38006590 con 0x7fec38002840 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 monclient(hunting): my global_id is 154205 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec34004f10 con 0x7fec38002840 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1177556309 0 0) 0x7fec34004f10 con 0x7fec38002840 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec38006590 con 0x7fec38002840 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fec3c002720 con 0x7fec38002840 2024-09-18T12:03:36.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 10 monclient: dump: 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4185761397 0 0) 0x7fec38006590 con 0x7fec38002840 2024-09-18T12:03:36.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.053+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec3800a140 con 0x7fec38002840 2024-09-18T12:03:36.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.057+0000 7fec50ff9700 10 client.164112.objecter ms_handle_connect 0x7fec34038580 2024-09-18T12:03:36.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.057+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fec3c002980 con 0x7fec38002840 2024-09-18T12:03:36.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.057+0000 7fec50ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:36.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.057+0000 7fec50ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:36.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.057+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6790/0 conn(0x7fec38002840 legacy=0x7fec38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.057+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6790/0 conn(0x7fec38002840 legacy=0x7fec38004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:36.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.065+0000 7fec5944d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.065+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fec38009670 con 0x7fec380090e0 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.065+0000 7fec50ff9700 10 client.164112.objecter ms_handle_connect 0x7fec380090e0 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.065+0000 7fec50ff9700 10 client.164112.objecter resend_mon_ops 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.065+0000 7fec4b7fe700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6790/0 conn(0x7fec380090e0 legacy=0x7fec38008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455774058 0 0) 0x7fec38009670 con 0x7fec380090e0 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 10 monclient(hunting): my global_id is 154208 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec38006590 con 0x7fec380090e0 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3112084804 0 0) 0x7fec38006590 con 0x7fec380090e0 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec38009670 con 0x7fec380090e0 2024-09-18T12:03:36.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fec3c0059b0 con 0x7fec380090e0 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 10 monclient: dump: 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 372283962 0 0) 0x7fec38009670 con 0x7fec380090e0 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 --> v1:172.21.15.112:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fec38009930 con 0x7fec380090e0 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 <== mon.2 v1:172.21.15.112:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4171 (unknown 0 0 2434736337) 0x7fec3c007b50 con 0x7fec380090e0 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6790/0 conn(0x7fec380090e0 legacy=0x7fec38008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec50ff9700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6790/0 conn(0x7fec380090e0 legacy=0x7fec38008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:36.081 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 82, 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:03:36.082 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:36.083 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:03:36.084 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:03:36.085 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:03:36.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.088 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:03:36.090 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:03:36.091 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T12:03:36.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": [ 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:03:36.093 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 10 librados: watch_flush enter 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 10 librados: watch_flush exit 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 20 client.164112.objecter shutdown clearing up homeless session... 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 10 client.164112.objecter successfully canceled tick 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.129:6812/20806 conn(0x7fec34038580 legacy=0x7fec3403aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 10 monclient: shutdown 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 >> v1:172.21.15.112:6789/0 conn(0x7fec5406c280 legacy=0x7fec541412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec58c4c700 1 -- 172.21.15.112:0/4250247642 reap_dead start 2024-09-18T12:03:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec58c4c700 1 -- 172.21.15.112:0/4250247642 reap_dead start 2024-09-18T12:03:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 shutdown_connections 2024-09-18T12:03:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 >> 172.21.15.112:0/4250247642 conn(0x7fec54067da0 msgr2=0x7fec54099f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 shutdown_connections 2024-09-18T12:03:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 1 -- 172.21.15.112:0/4250247642 wait complete. 2024-09-18T12:03:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.069+0000 7fec5944d700 1 librados: shutdown 2024-09-18T12:03:36.096 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:03:36.096 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:03:36.096 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:03:36.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient: get_monmap_and_config 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient: build_initial_monmap 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient: monmap: 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:36.209838+0000 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:36.209838+0000 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:36.209 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 1 Processor -- start 2024-09-18T12:03:36.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 1 -- start start 2024-09-18T12:03:36.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient: init 2024-09-18T12:03:36.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:36.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:03:36.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient(hunting): picked mon.noname-b con 0x7fc800130420 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient(hunting): picked mon.noname-a con 0x7fc80006ea60 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient(hunting): picked mon.noname-c con 0x7fc800136aa0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc800130f00 con 0x7fc80006ea60 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80006c990 con 0x7fc800136aa0 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8000a12c0 con 0x7fc800130420 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc8051c7700 10 monclient(hunting): authenticate will time out at 1988.050886s 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc7fed9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc800130420 0x7fc8001307f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35600/0 (socket says 172.21.15.112:35600) 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc7fed9d700 1 -- 172.21.15.112:0/2680652054 learned_addr learned my addr 172.21.15.112:0/2680652054 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:36.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.205+0000 7fc7fe59c700 1 --1- 172.21.15.112:0/2680652054 >> v1:172.21.15.112:6789/0 conn(0x7fc80006ea60 0x7fc800135380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51922/0 (socket says 172.21.15.112:51922) 2024-09-18T12:03:36.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fed9d700 1 -- 172.21.15.112:0/2680652054 >> v1:172.21.15.112:6790/0 conn(0x7fc800130420 legacy=0x7fc8001307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fe59c700 1 -- 172.21.15.112:0/2680652054 >> v1:172.21.15.112:6789/0 conn(0x7fc80006ea60 legacy=0x7fc800135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7ff59e700 1 -- 172.21.15.112:0/2680652054 >> v1:172.21.15.129:6789/0 conn(0x7fc800136aa0 legacy=0x7fc800138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315076203 0 0) 0x7fc800130f00 con 0x7fc80006ea60 2024-09-18T12:03:36.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): my global_id is 164115 2024-09-18T12:03:36.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e8003570 con 0x7fc80006ea60 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805148593 0 0) 0x7fc8000a12c0 con 0x7fc800130420 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): my global_id is 154211 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc800130f00 con 0x7fc800130420 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234291244 0 0) 0x7fc80006c990 con 0x7fc800136aa0 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient(hunting): my global_id is 144292 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8000a12c0 con 0x7fc800136aa0 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833787 0 0) 0x7fc7e8003570 con 0x7fc80006ea60 2024-09-18T12:03:36.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 >> v1:172.21.15.112:6790/0 conn(0x7fc800130420 legacy=0x7fc8001307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 >> v1:172.21.15.129:6789/0 conn(0x7fc800136aa0 legacy=0x7fc800138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: _finish_hunting 0 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 monclient: found mon.noname-a 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80006d7d0 con 0x7fc80006ea60 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: _finish_auth 0 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 5 monclient: authenticate success, global_id 164115 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc7f4001940 con 0x7fc80006ea60 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: dump: 2024-09-18T12:03:36.217 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7f4001c10 con 0x7fc80006ea60 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:36.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 1 -- 172.21.15.112:0/2680652054 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc7f40049d0 con 0x7fc80006ea60 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 4 set_mon_vals no callback set 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc7fdd9b700 10 monclient: dump: 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.219 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.220 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 10 monclient: shutdown 2024-09-18T12:03:36.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:36.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 1 -- 172.21.15.112:0/2680652054 >> v1:172.21.15.112:6789/0 conn(0x7fc80006ea60 legacy=0x7fc800135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 1 -- 172.21.15.112:0/2680652054 shutdown_connections 2024-09-18T12:03:36.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 1 -- 172.21.15.112:0/2680652054 >> 172.21.15.112:0/2680652054 conn(0x7fc8000680d0 msgr2=0x7fc8000a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:36.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 1 -- 172.21.15.112:0/2680652054 shutdown_connections 2024-09-18T12:03:36.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.209+0000 7fc8051c7700 1 -- 172.21.15.112:0/2680652054 wait complete. 2024-09-18T12:03:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient: build_initial_monmap 2024-09-18T12:03:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient: monmap: 2024-09-18T12:03:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 librados: starting msgr at 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 librados: starting objecter 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 Processor -- start 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 -- start start 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 librados: setting wanted keys 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 librados: calling monclient init 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient: init 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:36.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient(hunting): picked mon.noname-a con 0x7fc800130420 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient(hunting): picked mon.noname-b con 0x7fc800136aa0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient(hunting): picked mon.noname-c con 0x7fc80009b040 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80009bce0 con 0x7fc800130420 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc800136830 con 0x7fc800136aa0 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc7ff59e700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fc80009b040 0x7fc800147f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35610/0 (socket says 172.21.15.112:35610) 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc7ff59e700 1 -- 172.21.15.112:0/905842273 learned_addr learned my addr 172.21.15.112:0/905842273 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc800099710 con 0x7fc80009b040 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.213+0000 7fc8051c7700 10 monclient(hunting): authenticate will time out at 1988.059041s 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7fed9d700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.112:6789/0 conn(0x7fc800130420 legacy=0x7fc8000988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7fed9d700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.112:6789/0 conn(0x7fc800130420 legacy=0x7fc8000988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:03:36.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 client.?.objecter ms_handle_connect 0x7fc800130420 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7fed9d700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.112:6789/0 conn(0x7fc800130420 legacy=0x7fc8000988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7fed9d700 1 --1- 172.21.15.112:0/905842273 >> v1:172.21.15.112:6789/0 conn(0x7fc800130420 0x7fc8000988f0 :-1 s=OPENED pgs=700 cs=1 l=1).handle_message read tag failed 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7fed9d700 1 --1- 172.21.15.112:0/905842273 >> v1:172.21.15.112:6789/0 conn(0x7fc800130420 0x7fc8000988f0 :-1 s=OPENED pgs=700 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7ff59e700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.112:6790/0 conn(0x7fc80009b040 legacy=0x7fc800147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6789/0 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 client.?.objecter ms_handle_connect 0x7fc80009b040 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095262268 0 0) 0x7fc800099710 con 0x7fc80009b040 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): my global_id is 154214 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7fe59c700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6789/0 conn(0x7fc800136aa0 legacy=0x7fc800099000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ec004f10 con 0x7fc80009b040 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 client.?.objecter ms_handle_connect 0x7fc800136aa0 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4078928766 0 0) 0x7fc7ec004f10 con 0x7fc80009b040 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc800099710 con 0x7fc80009b040 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865095559 0 0) 0x7fc800136830 con 0x7fc800136aa0 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): my global_id is 144295 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ec004f10 con 0x7fc800136aa0 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc7f0001d10 con 0x7fc80009b040 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): dump: 2024-09-18T12:03:36.228 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 92847374 0 0) 0x7fc800099710 con 0x7fc80009b040 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6789/0 conn(0x7fc800136aa0 legacy=0x7fc800099000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.112:6789/0 conn(0x7fc800130420 legacy=0x7fc8000988f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:36.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 monclient: found mon.c 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80009abc0 con 0x7fc80009b040 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient: _finish_auth 0 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 5 monclient: authenticate success, global_id 154214 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 10 monclient: _renew_subs 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc800131bf0 con 0x7fc80009b040 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 10 client.154214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 10 monclient: _renew_subs 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.230 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc800134f40 con 0x7fc80009b040 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 1 librados: init done 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7f00021b0 con 0x7fc80009b040 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc7f0002500 con 0x7fc80009b040 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7fd59a700 4 set_mon_vals no callback set 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 10 monclient: dump: 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.231 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc7e0006590 con 0x7fc7e0002840 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.217+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fc7f0002760 con 0x7fc80009b040 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fc7f0043b10 con 0x7fc80009b040 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 client.154214.objecter ms_dispatch 0x7fc800000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 3 client.154214.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 3 client.154214.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 20 client.154214.objecter dump_active .. 0 homeless 2024-09-18T12:03:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7fe59c700 10 monclient: get_auth_request con 0x7fc7ec038190 auth_method 0 2024-09-18T12:03:36.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7fed9d700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6789/0 conn(0x7fc7e0002840 legacy=0x7fc7e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 client.154214.objecter ms_handle_connect 0x7fc7e0002840 2024-09-18T12:03:36.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 client.154214.objecter resend_mon_ops 2024-09-18T12:03:36.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504365925 0 0) 0x7fc7e0006590 con 0x7fc7e0002840 2024-09-18T12:03:36.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 monclient(hunting): my global_id is 144298 2024-09-18T12:03:36.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc800099710 con 0x7fc7e0002840 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 628282497 0 0) 0x7fc800099710 con 0x7fc7e0002840 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7e0006590 con 0x7fc7e0002840 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc7f8001a20 con 0x7fc7e0002840 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 monclient: dump: 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 10 client.154214.objecter ms_handle_connect 0x7fc7ec038190 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 321107734 0 0) 0x7fc7e0006590 con 0x7fc7e0002840 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.221+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7e000a140 con 0x7fc7e0002840 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.225+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fc7f8004bb0 con 0x7fc7e0002840 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.225+0000 7fc7e6ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.225+0000 7fc7e6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.225+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6789/0 conn(0x7fc7e0002840 legacy=0x7fc7e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.225+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6789/0 conn(0x7fc7e0002840 legacy=0x7fc7e0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:36.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.233+0000 7fc8051c7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:03:36.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.233+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc7e0007550 con 0x7fc7e0006900 2024-09-18T12:03:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7fed9d700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6789/0 conn(0x7fc7e0006900 legacy=0x7fc7e0006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 client.154214.objecter ms_handle_connect 0x7fc7e0006900 2024-09-18T12:03:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 client.154214.objecter resend_mon_ops 2024-09-18T12:03:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117925747 0 0) 0x7fc7e0007550 con 0x7fc7e0006900 2024-09-18T12:03:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 monclient(hunting): my global_id is 144301 2024-09-18T12:03:36.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e0006590 con 0x7fc7e0006900 2024-09-18T12:03:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3360994234 0 0) 0x7fc7e0006590 con 0x7fc7e0006900 2024-09-18T12:03:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7e0007550 con 0x7fc7e0006900 2024-09-18T12:03:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fc7f80051f0 con 0x7fc7e0006900 2024-09-18T12:03:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 monclient: dump: 2024-09-18T12:03:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.241 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1130582275 0 0) 0x7fc7e0007550 con 0x7fc7e0006900 2024-09-18T12:03:36.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 --> v1:172.21.15.129:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc7e00087b0 con 0x7fc7e0006900 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 <== mon.1 v1:172.21.15.129:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4001 (unknown 0 0 4158912248) 0x7fc7f8007430 con 0x7fc7e0006900 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6789/0 conn(0x7fc7e0006900 legacy=0x7fc7e0006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc7e6ffd700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6789/0 conn(0x7fc7e0006900 legacy=0x7fc7e0006d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "peon", 2024-09-18T12:03:36.243 INFO:teuthology.orchestra.run.smithi112.stdout: "election_epoch": 82, 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum": [ 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: 0, 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: 1, 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: 2 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_age": 0, 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "required_con": "2449958755906961412", 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "required_mon": [ 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:36.244 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "quorum_mon": [ 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:36.245 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "outside_quorum": [], 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "extra_probe_peers": [], 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_provider": [], 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "monmap": { 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "fsid": "e68e10b8-0874-4c25-87a0-f38d9ca2e0d2", 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "modified": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "created": "2024-09-18T11:43:39.500070Z", 2024-09-18T12:03:36.246 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release": 17, 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "election_strategy": 1, 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "disallowed_leaders: ": "", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false, 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "tiebreaker_mon": "", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "removed_ranks: ": "", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "features": { 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "persistent": [ 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "kraken", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "luminous", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "mimic", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap-prune", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "nautilus", 2024-09-18T12:03:36.247 INFO:teuthology.orchestra.run.smithi112.stdout: "octopus", 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "pacific", 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "elector-pinging", 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "quincy" 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "optional": [] 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "mons": [ 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 0, 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.248 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789", 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6789/0", 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6789/0", 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 1, 2024-09-18T12:03:36.249 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789", 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.129:6789/0", 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.129:6789/0", 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:36.250 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: "rank": 2, 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "c", 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addrs": { 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: "addrvec": [ 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790", 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 0 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "172.21.15.112:6790/0", 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "public_addr": "172.21.15.112:6790/0", 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "weight": 0, 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "crush_location": "{}" 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "feature_map": { 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": [ 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:36.252 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 1 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": [ 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 2 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: "client": [ 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:03:36.253 INFO:teuthology.orchestra.run.smithi112.stdout: "release": "luminous", 2024-09-18T12:03:36.254 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4 2024-09-18T12:03:36.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-18T12:03:36.254 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-18T12:03:36.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-18T12:03:36.254 INFO:teuthology.orchestra.run.smithi112.stdout: "stretch_mode": false 2024-09-18T12:03:36.254 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:36.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc8051c7700 10 librados: watch_flush enter 2024-09-18T12:03:36.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc8051c7700 10 librados: watch_flush exit 2024-09-18T12:03:36.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc8051c7700 20 client.154214.objecter shutdown clearing up homeless session... 2024-09-18T12:03:36.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.237+0000 7fc8051c7700 10 client.154214.objecter successfully canceled tick 2024-09-18T12:03:36.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.129:6812/20806 conn(0x7fc7ec038190 legacy=0x7fc7ec03a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 10 monclient: shutdown 2024-09-18T12:03:36.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:36.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 >> v1:172.21.15.112:6790/0 conn(0x7fc80009b040 legacy=0x7fc800147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc7ff59e700 1 -- 172.21.15.112:0/905842273 reap_dead start 2024-09-18T12:03:36.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc7ff59e700 1 -- 172.21.15.112:0/905842273 reap_dead start 2024-09-18T12:03:36.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 shutdown_connections 2024-09-18T12:03:36.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 >> 172.21.15.112:0/905842273 conn(0x7fc8000680d0 msgr2=0x7fc80009dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:36.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 shutdown_connections 2024-09-18T12:03:36.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 1 -- 172.21.15.112:0/905842273 wait complete. 2024-09-18T12:03:36.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.241+0000 7fc8051c7700 1 librados: shutdown 2024-09-18T12:03:36.260 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:03:36.260 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:03:36.260 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:03:36.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient: get_monmap_and_config 2024-09-18T12:03:36.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient: build_initial_monmap 2024-09-18T12:03:36.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient: monmap: 2024-09-18T12:03:36.375 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:36.376633+0000 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:36.376633+0000 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:36.376 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 1 Processor -- start 2024-09-18T12:03:36.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 1 -- start start 2024-09-18T12:03:36.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient: init 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient(hunting): picked mon.noname-c con 0x7f7b60068690 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient(hunting): picked mon.noname-b con 0x7f7b6006e1d0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient(hunting): picked mon.noname-a con 0x7f7b60097a50 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b60097350 con 0x7f7b60097a50 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6006d570 con 0x7f7b60068690 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b60097f70 con 0x7f7b6006e1d0 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b65ead700 10 monclient(hunting): authenticate will time out at 1988.217525s 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b5ffff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f7b6006e1d0 0x7f7b6006ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35618/0 (socket says 172.21.15.112:35618) 2024-09-18T12:03:36.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b5ffff700 1 -- 172.21.15.112:0/3239332759 learned_addr learned my addr 172.21.15.112:0/3239332759 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.373+0000 7f7b5ffff700 1 -- 172.21.15.112:0/3239332759 >> v1:172.21.15.112:6790/0 conn(0x7f7b6006e1d0 legacy=0x7f7b6006ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b656ac700 1 -- 172.21.15.112:0/3239332759 >> v1:172.21.15.112:6789/0 conn(0x7f7b60097a50 legacy=0x7f7b60138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b64eab700 1 -- 172.21.15.112:0/3239332759 >> v1:172.21.15.129:6789/0 conn(0x7f7b60068690 legacy=0x7f7b6006c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679180040 0 0) 0x7f7b60097f70 con 0x7f7b6006e1d0 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): my global_id is 154217 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b48003570 con 0x7f7b6006e1d0 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760259807 0 0) 0x7f7b60097350 con 0x7f7b60097a50 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.382 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): my global_id is 164118 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b60097f70 con 0x7f7b60097a50 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198963147 0 0) 0x7f7b6006d570 con 0x7f7b60068690 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient(hunting): my global_id is 144304 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b60097350 con 0x7f7b60068690 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 917038558 0 0) 0x7f7b48003570 con 0x7f7b6006e1d0 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 >> v1:172.21.15.129:6789/0 conn(0x7f7b60068690 legacy=0x7f7b6006c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 >> v1:172.21.15.112:6789/0 conn(0x7f7b60097a50 legacy=0x7f7b60138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 monclient: found mon.noname-b 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:36.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6006ed70 con 0x7f7b6006e1d0 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: _finish_auth 0 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7b50001cc0 con 0x7f7b6006e1d0 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: dump: 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.384 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 5 monclient: authenticate success, global_id 154217 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b50001f50 con 0x7f7b6006e1d0 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 4 set_mon_vals no callback set 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 1 -- 172.21.15.112:0/3239332759 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7b50004e10 con 0x7f7b6006e1d0 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b5f7fe700 10 monclient: dump: 2024-09-18T12:03:36.385 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient: shutdown 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- 172.21.15.112:0/3239332759 >> v1:172.21.15.112:6790/0 conn(0x7f7b6006e1d0 legacy=0x7f7b6006ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- 172.21.15.112:0/3239332759 shutdown_connections 2024-09-18T12:03:36.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- 172.21.15.112:0/3239332759 >> 172.21.15.112:0/3239332759 conn(0x7f7b600a01e0 msgr2=0x7f7b600a05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- 172.21.15.112:0/3239332759 shutdown_connections 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- 172.21.15.112:0/3239332759 wait complete. 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient: build_initial_monmap 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient: monmap: 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 librados: starting msgr at 2024-09-18T12:03:36.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 librados: starting objecter 2024-09-18T12:03:36.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 Processor -- start 2024-09-18T12:03:36.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- start start 2024-09-18T12:03:36.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 librados: setting wanted keys 2024-09-18T12:03:36.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 librados: calling monclient init 2024-09-18T12:03:36.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient: init 2024-09-18T12:03:36.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:36.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient(hunting): picked mon.noname-c con 0x7f7b60068690 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient(hunting): picked mon.noname-a con 0x7f7b6006e1d0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient(hunting): picked mon.noname-b con 0x7f7b60097a50 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b60123170 con 0x7f7b6006e1d0 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6009f080 con 0x7f7b60097a50 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6006ef70 con 0x7f7b60068690 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b65ead700 10 monclient(hunting): authenticate will time out at 1988.223474s 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b64eab700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f7b60068690 0x7f7b600698c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35626/0 (socket says 172.21.15.112:35626) 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.377+0000 7f7b64eab700 1 -- 172.21.15.112:0/4233626069 learned_addr learned my addr 172.21.15.112:0/4233626069 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b64eab700 1 -- 172.21.15.112:0/4233626069 >> v1:172.21.15.112:6790/0 conn(0x7f7b60068690 legacy=0x7f7b600698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 client.?.objecter ms_handle_connect 0x7f7b60068690 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 client.?.objecter ms_handle_connect 0x7f7b6006e1d0 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5ffff700 1 -- 172.21.15.112:0/4233626069 >> v1:172.21.15.112:6789/0 conn(0x7f7b6006e1d0 legacy=0x7f7b60069fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b656ac700 1 -- 172.21.15.112:0/4233626069 >> v1:172.21.15.129:6789/0 conn(0x7f7b60097a50 legacy=0x7f7b6006a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 client.?.objecter ms_handle_connect 0x7f7b60097a50 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146607741 0 0) 0x7f7b6006ef70 con 0x7f7b60068690 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): my global_id is 154220 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b4c004f10 con 0x7f7b60068690 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451825249 0 0) 0x7f7b6006ed70 con 0x7f7b6006e1d0 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): my global_id is 164121 2024-09-18T12:03:36.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6006ef70 con 0x7f7b6006e1d0 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638931806 0 0) 0x7f7b6009f080 con 0x7f7b60097a50 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): my global_id is 144307 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6006ed70 con 0x7f7b60097a50 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898532495 0 0) 0x7f7b4c004f10 con 0x7f7b60068690 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6009f080 con 0x7f7b60068690 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7b54001c30 con 0x7f7b60068690 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): dump: 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.395 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1951315980 0 0) 0x7f7b6006ef70 con 0x7f7b6006e1d0 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b4c004f10 con 0x7f7b6006e1d0 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3009245300 0 0) 0x7f7b6006ed70 con 0x7f7b60097a50 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6006ef70 con 0x7f7b60097a50 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7b500016e0 con 0x7f7b6006e1d0 2024-09-18T12:03:36.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): dump: 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7b58001c40 con 0x7f7b60097a50 2024-09-18T12:03:36.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient(hunting): dump: 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 73262782 0 0) 0x7f7b6009f080 con 0x7f7b60068690 2024-09-18T12:03:36.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 >> v1:172.21.15.129:6789/0 conn(0x7f7b60097a50 legacy=0x7f7b6006a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 >> v1:172.21.15.112:6789/0 conn(0x7f7b6006e1d0 legacy=0x7f7b60069fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient: _finish_hunting 0 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 monclient: found mon.c 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b601389d0 con 0x7f7b60068690 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient: _finish_auth 0 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 5 monclient: authenticate success, global_id 154220 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 10 monclient: _renew_subs 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b601210a0 con 0x7f7b60068690 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 10 client.154220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 10 monclient: _renew_subs 2024-09-18T12:03:36.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b60121510 con 0x7f7b60068690 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b65ead700 1 librados: init done 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b540022b0 con 0x7f7b60068690 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f7b54002680 con 0x7f7b60068690 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5cff9700 10 monclient: dump: 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:36.400 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.381+0000 7f7b5effd700 4 set_mon_vals no callback set 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b65ead700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b65ead700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b65ead700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f7b540027e0 con 0x7f7b60068690 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b400050b0 con 0x7f7b60068690 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f7b54043c80 con 0x7f7b60068690 2024-09-18T12:03:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b5cff9700 10 client.154220.objecter ms_dispatch 0x7f7b60000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b5cff9700 3 client.154220.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b5cff9700 3 client.154220.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b5cff9700 20 client.154220.objecter dump_active .. 0 homeless 2024-09-18T12:03:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.385+0000 7f7b5ffff700 10 monclient: get_auth_request con 0x7f7b4c0385b0 auth_method 0 2024-09-18T12:03:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.389+0000 7f7b5cff9700 10 client.154220.objecter ms_handle_connect 0x7f7b4c0385b0 2024-09-18T12:03:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.393+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f7b54014b40 con 0x7f7b60068690 2024-09-18T12:03:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.393+0000 7f7b5cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.393+0000 7f7b5cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:36.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.629+0000 7f7b65ead700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:03:36.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.629+0000 7f7b65ead700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:03:36.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.629+0000 7f7b65ead700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:36.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.629+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7b40005c30 con 0x7f7b60068690 2024-09-18T12:03:36.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b5cff9700 1 -- 172.21.15.112:0/4233626069 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f7b54014540 con 0x7f7b60068690 2024-09-18T12:03:36.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b5cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:03:36.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b5cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:03:36.636 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.636 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=8,logm=55} crc {auth=2178005844,config=3140514009,health=2077692657,kv=1310296871,logm=2046940326}) 2024-09-18T12:03:36.636 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=7,logm=56} crc {auth=2106407121,config=3140514009,health=1310840331,kv=1310296871,logm=2586038374}) 2024-09-18T12:03:36.636 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.637 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=8,logm=55} crc {auth=2178005844,config=3140514009,health=2077692657,kv=1310296871,logm=2046940326}) 2024-09-18T12:03:36.637 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=7,logm=55} crc {auth=2603026353,config=3140514009,health=2077692657,kv=3173730989,logm=3245784767}) 2024-09-18T12:03:36.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 10 librados: watch_flush enter 2024-09-18T12:03:36.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 10 librados: watch_flush exit 2024-09-18T12:03:36.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 20 client.154220.objecter shutdown clearing up homeless session... 2024-09-18T12:03:36.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 10 client.154220.objecter successfully canceled tick 2024-09-18T12:03:36.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 >> v1:172.21.15.129:6812/20806 conn(0x7f7b4c0385b0 legacy=0x7f7b4c03aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 10 monclient: shutdown 2024-09-18T12:03:36.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:36.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 >> v1:172.21.15.112:6790/0 conn(0x7f7b60068690 legacy=0x7f7b600698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:36.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 shutdown_connections 2024-09-18T12:03:36.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 >> 172.21.15.112:0/4233626069 conn(0x7f7b600a01e0 msgr2=0x7f7b600a14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:36.639 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.639 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277212805}) 2024-09-18T12:03:36.639 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1066068622}) 2024-09-18T12:03:36.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 shutdown_connections 2024-09-18T12:03:36.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 1 -- 172.21.15.112:0/4233626069 wait complete. 2024-09-18T12:03:36.640 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.640 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277212805}) 2024-09-18T12:03:36.640 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777166571}) 2024-09-18T12:03:36.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:36.633+0000 7f7b65ead700 1 librados: shutdown 2024-09-18T12:03:36.642 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.637+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.642 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.637+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972996669}) 2024-09-18T12:03:36.642 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.637+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889118282}) 2024-09-18T12:03:36.642 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.637+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.642 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.637+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972996669}) 2024-09-18T12:03:36.642 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.637+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31005420}) 2024-09-18T12:03:36.646 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.641+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.646 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.641+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731736666}) 2024-09-18T12:03:36.646 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.641+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40355702}) 2024-09-18T12:03:36.646 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.641+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.646 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.641+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731736666}) 2024-09-18T12:03:36.646 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.641+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705102031}) 2024-09-18T12:03:36.652 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.649+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.652 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.649+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3021730274}) 2024-09-18T12:03:36.652 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.649+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1261269051}) 2024-09-18T12:03:36.652 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.649+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.652 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.649+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3021730274}) 2024-09-18T12:03:36.652 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.649+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057026975}) 2024-09-18T12:03:36.653 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:03:36.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1725830922}) 2024-09-18T12:03:36.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=545794278}) 2024-09-18T12:03:36.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1725830922}) 2024-09-18T12:03:36.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3243100168}) 2024-09-18T12:03:36.657 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866340749}) 2024-09-18T12:03:36.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1215678609}) 2024-09-18T12:03:36.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866340749}) 2024-09-18T12:03:36.658 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4049315998}) 2024-09-18T12:03:36.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3509736374}) 2024-09-18T12:03:36.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2552137583}) 2024-09-18T12:03:36.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3509736374}) 2024-09-18T12:03:36.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069660133}) 2024-09-18T12:03:36.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497354553}) 2024-09-18T12:03:36.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271664786}) 2024-09-18T12:03:36.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497354553}) 2024-09-18T12:03:36.661 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3198987097}) 2024-09-18T12:03:36.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138583009}) 2024-09-18T12:03:36.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1348440191}) 2024-09-18T12:03:36.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138583009}) 2024-09-18T12:03:36.662 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.657+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3676155961}) 2024-09-18T12:03:36.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.661+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.661+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694419008}) 2024-09-18T12:03:36.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.661+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=496113145}) 2024-09-18T12:03:36.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.661+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.661+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694419008}) 2024-09-18T12:03:36.664 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.661+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1976246548}) 2024-09-18T12:03:36.675 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.676 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217432568}) 2024-09-18T12:03:36.676 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084856363}) 2024-09-18T12:03:36.676 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.676 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217432568}) 2024-09-18T12:03:36.676 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464108270}) 2024-09-18T12:03:36.677 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2532584974}) 2024-09-18T12:03:36.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2237988552}) 2024-09-18T12:03:36.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2532584974}) 2024-09-18T12:03:36.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1590337565}) 2024-09-18T12:03:36.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.773+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.773+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4054036094}) 2024-09-18T12:03:36.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.773+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=606922093}) 2024-09-18T12:03:36.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.773+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.773+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4054036094}) 2024-09-18T12:03:36.778 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.773+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=744384730}) 2024-09-18T12:03:36.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027144744}) 2024-09-18T12:03:36.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318738573}) 2024-09-18T12:03:36.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.875 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027144744}) 2024-09-18T12:03:36.876 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=597927943}) 2024-09-18T12:03:36.877 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931134789}) 2024-09-18T12:03:36.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2430699263}) 2024-09-18T12:03:36.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931134789}) 2024-09-18T12:03:36.878 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.873+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605792816}) 2024-09-18T12:03:36.880 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.880 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969359345}) 2024-09-18T12:03:36.880 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1903621835}) 2024-09-18T12:03:36.880 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.880 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969359345}) 2024-09-18T12:03:36.880 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2510149451}) 2024-09-18T12:03:36.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=164065093}) 2024-09-18T12:03:36.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297970469}) 2024-09-18T12:03:36.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=164065093}) 2024-09-18T12:03:36.882 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.877+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404619967}) 2024-09-18T12:03:36.884 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.884 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233168912}) 2024-09-18T12:03:36.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=985844224}) 2024-09-18T12:03:36.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233168912}) 2024-09-18T12:03:36.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787398234}) 2024-09-18T12:03:36.885 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.886 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250880201}) 2024-09-18T12:03:36.886 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4252143299}) 2024-09-18T12:03:36.886 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.886 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250880201}) 2024-09-18T12:03:36.886 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.881+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2924111666}) 2024-09-18T12:03:36.887 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.887 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170619548}) 2024-09-18T12:03:36.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2368442468}) 2024-09-18T12:03:36.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170619548}) 2024-09-18T12:03:36.888 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3389196864}) 2024-09-18T12:03:36.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305840837}) 2024-09-18T12:03:36.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099092104}) 2024-09-18T12:03:36.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.889 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305840837}) 2024-09-18T12:03:36.890 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=495982957}) 2024-09-18T12:03:36.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202413166}) 2024-09-18T12:03:36.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1279707893}) 2024-09-18T12:03:36.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202413166}) 2024-09-18T12:03:36.891 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.885+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2381383723}) 2024-09-18T12:03:36.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701467713}) 2024-09-18T12:03:36.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=742195870}) 2024-09-18T12:03:36.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701467713}) 2024-09-18T12:03:36.893 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=962218994}) 2024-09-18T12:03:36.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965942180}) 2024-09-18T12:03:36.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551281231}) 2024-09-18T12:03:36.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965942180}) 2024-09-18T12:03:36.894 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.889+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4198036282}) 2024-09-18T12:03:36.895 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.895 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527155351}) 2024-09-18T12:03:36.895 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863272373}) 2024-09-18T12:03:36.896 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.896 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527155351}) 2024-09-18T12:03:36.896 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1498602768}) 2024-09-18T12:03:36.896 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.896 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313414907}) 2024-09-18T12:03:36.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=765863451}) 2024-09-18T12:03:36.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313414907}) 2024-09-18T12:03:36.897 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=895677029}) 2024-09-18T12:03:36.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450598464}) 2024-09-18T12:03:36.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752244308}) 2024-09-18T12:03:36.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450598464}) 2024-09-18T12:03:36.898 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.893+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371965516}) 2024-09-18T12:03:36.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913483169}) 2024-09-18T12:03:36.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1206202785}) 2024-09-18T12:03:36.899 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913483169}) 2024-09-18T12:03:36.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975123011}) 2024-09-18T12:03:36.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696719830}) 2024-09-18T12:03:36.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709388872}) 2024-09-18T12:03:36.900 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.901 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696719830}) 2024-09-18T12:03:36.901 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=36520556}) 2024-09-18T12:03:36.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562588751}) 2024-09-18T12:03:36.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1155102953}) 2024-09-18T12:03:36.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562588751}) 2024-09-18T12:03:36.902 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.897+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376571818}) 2024-09-18T12:03:36.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177946803}) 2024-09-18T12:03:36.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2260429187}) 2024-09-18T12:03:36.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.903 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177946803}) 2024-09-18T12:03:36.904 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208504943}) 2024-09-18T12:03:36.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291591744}) 2024-09-18T12:03:36.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=21785866}) 2024-09-18T12:03:36.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291591744}) 2024-09-18T12:03:36.905 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170036260}) 2024-09-18T12:03:36.906 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.906 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229178148}) 2024-09-18T12:03:36.906 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1315018068}) 2024-09-18T12:03:36.906 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.906 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229178148}) 2024-09-18T12:03:36.906 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.901+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3351798290}) 2024-09-18T12:03:36.908 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.908 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578885172}) 2024-09-18T12:03:36.908 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3896604448}) 2024-09-18T12:03:36.908 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.908 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578885172}) 2024-09-18T12:03:36.908 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232907555}) 2024-09-18T12:03:36.910 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.910 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=502592778}) 2024-09-18T12:03:36.910 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3533064156}) 2024-09-18T12:03:36.910 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.910 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=502592778}) 2024-09-18T12:03:36.910 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.905+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2432989646}) 2024-09-18T12:03:36.912 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.912 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067841679}) 2024-09-18T12:03:36.912 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1048326530}) 2024-09-18T12:03:36.912 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.912 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067841679}) 2024-09-18T12:03:36.912 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1725549168}) 2024-09-18T12:03:36.914 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.914 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292744676}) 2024-09-18T12:03:36.914 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3703156392}) 2024-09-18T12:03:36.914 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.914 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292744676}) 2024-09-18T12:03:36.914 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.909+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1410181041}) 2024-09-18T12:03:36.915 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.915 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=197952804}) 2024-09-18T12:03:36.915 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294867874}) 2024-09-18T12:03:36.915 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.916 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=197952804}) 2024-09-18T12:03:36.916 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964481027}) 2024-09-18T12:03:36.917 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.917 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763132901}) 2024-09-18T12:03:36.917 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=205777978}) 2024-09-18T12:03:36.917 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.917 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763132901}) 2024-09-18T12:03:36.917 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3584487859}) 2024-09-18T12:03:36.918 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.918 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439228892}) 2024-09-18T12:03:36.918 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1022151675}) 2024-09-18T12:03:36.918 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.919 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439228892}) 2024-09-18T12:03:36.919 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.913+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1677522087}) 2024-09-18T12:03:36.920 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.920 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1221268164}) 2024-09-18T12:03:36.920 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=493094049}) 2024-09-18T12:03:36.920 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.920 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1221268164}) 2024-09-18T12:03:36.920 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2242572486}) 2024-09-18T12:03:36.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3898647119}) 2024-09-18T12:03:36.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3569173102}) 2024-09-18T12:03:36.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3898647119}) 2024-09-18T12:03:36.921 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286897617}) 2024-09-18T12:03:36.922 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414722519}) 2024-09-18T12:03:36.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4218240867}) 2024-09-18T12:03:36.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414722519}) 2024-09-18T12:03:36.923 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.917+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2637674011}) 2024-09-18T12:03:36.934 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.929+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.934 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.929+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490818613}) 2024-09-18T12:03:36.934 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.929+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1511148096}) 2024-09-18T12:03:36.934 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.929+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.934 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.929+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490818613}) 2024-09-18T12:03:36.934 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.929+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3032099217}) 2024-09-18T12:03:36.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504144773}) 2024-09-18T12:03:36.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2360736031}) 2024-09-18T12:03:36.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504144773}) 2024-09-18T12:03:36.936 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602255091}) 2024-09-18T12:03:36.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1136359936}) 2024-09-18T12:03:36.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857555997}) 2024-09-18T12:03:36.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1136359936}) 2024-09-18T12:03:36.938 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.933+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2285965609}) 2024-09-18T12:03:36.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646853866}) 2024-09-18T12:03:36.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915001294}) 2024-09-18T12:03:36.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646853866}) 2024-09-18T12:03:36.940 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1122757980}) 2024-09-18T12:03:36.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749486142}) 2024-09-18T12:03:36.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1139460716}) 2024-09-18T12:03:36.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749486142}) 2024-09-18T12:03:36.942 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.937+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613625479}) 2024-09-18T12:03:36.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149969110}) 2024-09-18T12:03:36.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3255804116}) 2024-09-18T12:03:36.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149969110}) 2024-09-18T12:03:36.944 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376443541}) 2024-09-18T12:03:36.945 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.946 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818264467}) 2024-09-18T12:03:36.946 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497283719}) 2024-09-18T12:03:36.946 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.946 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818264467}) 2024-09-18T12:03:36.946 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.941+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631475539}) 2024-09-18T12:03:36.947 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.947 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565064634}) 2024-09-18T12:03:36.947 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=458464353}) 2024-09-18T12:03:36.947 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.947 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565064634}) 2024-09-18T12:03:36.948 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2338322021}) 2024-09-18T12:03:36.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120291665}) 2024-09-18T12:03:36.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4078180841}) 2024-09-18T12:03:36.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120291665}) 2024-09-18T12:03:36.949 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=942271525}) 2024-09-18T12:03:36.950 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.950 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044399174}) 2024-09-18T12:03:36.950 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2553224860}) 2024-09-18T12:03:36.950 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.950 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044399174}) 2024-09-18T12:03:36.951 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.945+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913279766}) 2024-09-18T12:03:36.951 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.952 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912185579}) 2024-09-18T12:03:36.952 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899709394}) 2024-09-18T12:03:36.952 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.952 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912185579}) 2024-09-18T12:03:36.952 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2707445849}) 2024-09-18T12:03:36.953 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.953 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016717529}) 2024-09-18T12:03:36.953 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3718292844}) 2024-09-18T12:03:36.953 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.953 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016717529}) 2024-09-18T12:03:36.953 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3766060969}) 2024-09-18T12:03:36.955 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.955 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840340654}) 2024-09-18T12:03:36.955 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2780796951}) 2024-09-18T12:03:36.955 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:36.955 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840340654}) 2024-09-18T12:03:36.955 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:36.949+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903884869}) 2024-09-18T12:03:37.186 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.181+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.186 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.181+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434713268}) 2024-09-18T12:03:37.186 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.181+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3400225804}) 2024-09-18T12:03:37.186 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.181+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.186 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.181+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434713268}) 2024-09-18T12:03:37.186 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.181+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3086586258}) 2024-09-18T12:03:37.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.521+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.521+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253683992}) 2024-09-18T12:03:37.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.521+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1365458303}) 2024-09-18T12:03:37.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.521+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.521+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253683992}) 2024-09-18T12:03:37.525 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.521+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174903474}) 2024-09-18T12:03:37.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.525+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.525+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805233506}) 2024-09-18T12:03:37.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.525+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899698783}) 2024-09-18T12:03:37.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.525+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.525+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805233506}) 2024-09-18T12:03:37.527 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.525+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=401104167}) 2024-09-18T12:03:37.654 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:03:37.654 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:03:37.655 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039179758}) 2024-09-18T12:03:37.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3231500122}) 2024-09-18T12:03:37.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039179758}) 2024-09-18T12:03:37.656 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2736775721}) 2024-09-18T12:03:37.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229714822}) 2024-09-18T12:03:37.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2570788727}) 2024-09-18T12:03:37.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229714822}) 2024-09-18T12:03:37.659 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.653+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188559377}) 2024-09-18T12:03:37.678 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.679 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576047405}) 2024-09-18T12:03:37.679 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4085749800}) 2024-09-18T12:03:37.679 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.679 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576047405}) 2024-09-18T12:03:37.679 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.673+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680649691}) 2024-09-18T12:03:37.680 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.680 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875604678}) 2024-09-18T12:03:37.680 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4075251842}) 2024-09-18T12:03:37.680 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.680 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875604678}) 2024-09-18T12:03:37.680 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2180217109}) 2024-09-18T12:03:37.681 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.681 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1226137953}) 2024-09-18T12:03:37.681 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337081134}) 2024-09-18T12:03:37.681 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.681 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1226137953}) 2024-09-18T12:03:37.682 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2432793370}) 2024-09-18T12:03:37.682 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.683 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=173066331}) 2024-09-18T12:03:37.683 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1871710956}) 2024-09-18T12:03:37.683 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.683 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=173066331}) 2024-09-18T12:03:37.683 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.677+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=956864741}) 2024-09-18T12:03:37.684 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.684 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050500703}) 2024-09-18T12:03:37.684 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868346336}) 2024-09-18T12:03:37.684 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.684 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050500703}) 2024-09-18T12:03:37.684 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3271998825}) 2024-09-18T12:03:37.685 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.685 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655893711}) 2024-09-18T12:03:37.685 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192403520}) 2024-09-18T12:03:37.685 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.685 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655893711}) 2024-09-18T12:03:37.686 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610403595}) 2024-09-18T12:03:37.686 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.686 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338550251}) 2024-09-18T12:03:37.686 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1287538688}) 2024-09-18T12:03:37.686 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.687 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338550251}) 2024-09-18T12:03:37.687 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.681+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1601491295}) 2024-09-18T12:03:37.687 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.687 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419787}) 2024-09-18T12:03:37.688 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350871949}) 2024-09-18T12:03:37.688 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.688 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419787}) 2024-09-18T12:03:37.688 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=517259971}) 2024-09-18T12:03:37.689 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.689 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005945694}) 2024-09-18T12:03:37.689 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=749200516}) 2024-09-18T12:03:37.689 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.689 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005945694}) 2024-09-18T12:03:37.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3240604172}) 2024-09-18T12:03:37.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134527310}) 2024-09-18T12:03:37.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1211357607}) 2024-09-18T12:03:37.690 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.691 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134527310}) 2024-09-18T12:03:37.691 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.685+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2636401369}) 2024-09-18T12:03:37.692 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.692 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830999666}) 2024-09-18T12:03:37.692 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070901065}) 2024-09-18T12:03:37.692 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.692 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830999666}) 2024-09-18T12:03:37.692 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2114620284}) 2024-09-18T12:03:37.693 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.693 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3979314904}) 2024-09-18T12:03:37.693 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3929286519}) 2024-09-18T12:03:37.693 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.693 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3979314904}) 2024-09-18T12:03:37.693 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2351422744}) 2024-09-18T12:03:37.694 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.694 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=321148313}) 2024-09-18T12:03:37.694 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=990084933}) 2024-09-18T12:03:37.694 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.694 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=321148313}) 2024-09-18T12:03:37.694 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=130523121}) 2024-09-18T12:03:37.695 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.695 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=6} crc {logm=3306818042,mdsmap=1531151339,mgr=315808716,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=324788651,mon_config_key=868353578,monmap=2808122079,osd_metadata=871764258,osd_pg_creating=1605287603,osd_snap=134652516}) 2024-09-18T12:03:37.695 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=58,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6} crc {logm=2075910435,mdsmap=1531151339,mgr=1454814743,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=323751850,mon_config_key=868353578,monmap=1725997317,osd_metadata=871764258,osd_pg_creating=1605287603,osd_snap=4134524712}) 2024-09-18T12:03:37.695 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.695 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=6} crc {logm=3306818042,mdsmap=1531151339,mgr=315808716,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=324788651,mon_config_key=868353578,monmap=2808122079,osd_metadata=871764258,osd_pg_creating=1605287603,osd_snap=134652516}) 2024-09-18T12:03:37.695 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.689+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=57,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3734391185,mdsmap=1531151339,mgr=2470090982,mgr_command_descs=47663135,mgr_metadata=454364145,mgrstat=2756831910,mon_config_key=868353578,monmap=2808122079,osd_metadata=871764258,osd_pg_creating=1605287603,osd_snap=3292239179}) 2024-09-18T12:03:37.696 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.696 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=34,osdmap=66} crc {osd_snap=30904295,osdmap=866362039}) 2024-09-18T12:03:37.696 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=37,osdmap=63} crc {osd_snap=2783894325,osdmap=1439891047}) 2024-09-18T12:03:37.696 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.696 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=34,osdmap=66} crc {osd_snap=30904295,osdmap=866362039}) 2024-09-18T12:03:37.696 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=36,osdmap=64} crc {osd_snap=624590272,osdmap=459328590}) 2024-09-18T12:03:37.697 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.697 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=573995624}) 2024-09-18T12:03:37.697 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4031608630}) 2024-09-18T12:03:37.698 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.698 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=573995624}) 2024-09-18T12:03:37.698 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3439926499}) 2024-09-18T12:03:37.699 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.699 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1443673053}) 2024-09-18T12:03:37.699 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3742450659}) 2024-09-18T12:03:37.699 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.699 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1443673053}) 2024-09-18T12:03:37.699 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.693+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1449312530}) 2024-09-18T12:03:37.700 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.701 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927286742}) 2024-09-18T12:03:37.701 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1485065815}) 2024-09-18T12:03:37.701 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.701 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=927286742}) 2024-09-18T12:03:37.701 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1806222163}) 2024-09-18T12:03:37.702 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.702 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2635647088}) 2024-09-18T12:03:37.702 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2341270678}) 2024-09-18T12:03:37.702 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.702 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2635647088}) 2024-09-18T12:03:37.702 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.697+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4063842094}) 2024-09-18T12:03:37.704 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.704 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=251472119}) 2024-09-18T12:03:37.704 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2582101958}) 2024-09-18T12:03:37.704 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.704 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=251472119}) 2024-09-18T12:03:37.704 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3650785533}) 2024-09-18T12:03:37.705 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.706 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2593202618}) 2024-09-18T12:03:37.706 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3964555076}) 2024-09-18T12:03:37.706 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.706 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2593202618}) 2024-09-18T12:03:37.706 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.701+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1792762124}) 2024-09-18T12:03:37.709 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.705+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.709 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.705+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=781499603}) 2024-09-18T12:03:37.709 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.705+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=295162768}) 2024-09-18T12:03:37.709 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.705+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.709 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.705+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=781499603}) 2024-09-18T12:03:37.709 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.705+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3047312234}) 2024-09-18T12:03:37.712 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.709+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.712 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.709+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1364296021}) 2024-09-18T12:03:37.712 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.709+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4132786399}) 2024-09-18T12:03:37.712 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.709+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.712 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.709+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1364296021}) 2024-09-18T12:03:37.712 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.709+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3734713033}) 2024-09-18T12:03:37.715 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.713+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.715 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.713+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2065812280}) 2024-09-18T12:03:37.715 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.713+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2824008024}) 2024-09-18T12:03:37.715 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.713+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.716 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.713+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2065812280}) 2024-09-18T12:03:37.716 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.713+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3912649414}) 2024-09-18T12:03:37.723 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.717+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.723 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.717+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=3897830393}) 2024-09-18T12:03:37.723 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.717+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=40} crc {osdmap=4285094209}) 2024-09-18T12:03:37.723 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.717+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:03:37.723 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.717+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=3897830393}) 2024-09-18T12:03:37.723 INFO:tasks.ceph.mon.a.smithi112.stderr:2024-09-18T12:03:37.717+0000 7f0009397700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=44} crc {osdmap=3580887321}) 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.753+0000 7f03a249c700 10 monclient: get_monmap_and_config 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.753+0000 7f03a249c700 10 monclient: build_initial_monmap 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.753+0000 7f03a249c700 10 monclient: monmap: 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:37.758366+0000 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:37.758366+0000 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:37.758 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.753+0000 7f03a249c700 1 Processor -- start 2024-09-18T12:03:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.753+0000 7f03a249c700 1 -- start start 2024-09-18T12:03:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.753+0000 7f03a249c700 10 monclient: init 2024-09-18T12:03:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 10 monclient(hunting): picked mon.noname-a con 0x7f039c06f150 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 10 monclient(hunting): picked mon.noname-c con 0x7f039c06f520 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 10 monclient(hunting): picked mon.noname-b con 0x7f039c06f920 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c0986a0 con 0x7f039c06f150 2024-09-18T12:03:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c09e560 con 0x7f039c06f520 2024-09-18T12:03:37.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c096b60 con 0x7f039c06f920 2024-09-18T12:03:37.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:37.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a249c700 10 monclient(hunting): authenticate will time out at 1989.600831s 2024-09-18T12:03:37.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a149a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f039c06f150 0x7f039c097040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51946/0 (socket says 172.21.15.112:51946) 2024-09-18T12:03:37.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a1c9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f039c06f920 0x7f039c136d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35634/0 (socket says 172.21.15.112:35634) 2024-09-18T12:03:37.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a149a700 1 -- 172.21.15.112:0/4119595839 learned_addr learned my addr 172.21.15.112:0/4119595839 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a149a700 1 -- 172.21.15.112:0/4119595839 >> v1:172.21.15.112:6789/0 conn(0x7f039c06f150 legacy=0x7f039c097040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a1c9b700 1 -- 172.21.15.112:0/4119595839 >> v1:172.21.15.112:6790/0 conn(0x7f039c06f920 legacy=0x7f039c136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f03a0c99700 1 -- 172.21.15.112:0/4119595839 >> v1:172.21.15.129:6789/0 conn(0x7f039c06f520 legacy=0x7f039c133620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866069671 0 0) 0x7f039c0986a0 con 0x7f039c06f150 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): my global_id is 164124 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0388003570 con 0x7f039c06f150 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32694968 0 0) 0x7f039c096b60 con 0x7f039c06f920 2024-09-18T12:03:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): my global_id is 154223 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f039c0986a0 con 0x7f039c06f920 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473701431 0 0) 0x7f039c09e560 con 0x7f039c06f520 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 10 monclient(hunting): my global_id is 144310 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.757+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f039c096b60 con 0x7f039c06f520 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3904225719 0 0) 0x7f0388003570 con 0x7f039c06f150 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 >> v1:172.21.15.112:6790/0 conn(0x7f039c06f920 legacy=0x7f039c136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 >> v1:172.21.15.129:6789/0 conn(0x7f039c06f520 legacy=0x7f039c133620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: _finish_hunting 0 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 1 monclient: found mon.noname-a 2024-09-18T12:03:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c098d80 con 0x7f039c06f150 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: _finish_auth 0 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0398001940 con 0x7f039c06f150 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: dump: 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 5 monclient: authenticate success, global_id 164124 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0398001c90 con 0x7f039c06f150 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 4 set_mon_vals no callback set 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 1 -- 172.21.15.112:0/4119595839 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0398004a90 con 0x7f039c06f150 2024-09-18T12:03:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f0393fff700 10 monclient: dump: 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 10 monclient: shutdown 2024-09-18T12:03:37.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 -- 172.21.15.112:0/4119595839 >> v1:172.21.15.112:6789/0 conn(0x7f039c06f150 legacy=0x7f039c097040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 -- 172.21.15.112:0/4119595839 shutdown_connections 2024-09-18T12:03:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 -- 172.21.15.112:0/4119595839 >> 172.21.15.112:0/4119595839 conn(0x7f039c0a1e30 msgr2=0x7f039c0a2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 -- 172.21.15.112:0/4119595839 shutdown_connections 2024-09-18T12:03:37.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 -- 172.21.15.112:0/4119595839 wait complete. 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 10 monclient: build_initial_monmap 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 10 monclient: monmap: 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 librados: starting msgr at 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 librados: starting objecter 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 Processor -- start 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 -- start start 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 librados: setting wanted keys 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 1 librados: calling monclient init 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.761+0000 7f03a249c700 10 monclient: init 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:37.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 10 monclient(hunting): picked mon.noname-c con 0x7f039c06f150 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 10 monclient(hunting): picked mon.noname-b con 0x7f039c06f520 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 10 monclient(hunting): picked mon.noname-a con 0x7f039c06f920 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c097870 con 0x7f039c06f920 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c09abc0 con 0x7f039c06f520 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c11b8b0 con 0x7f039c06f150 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a249c700 10 monclient(hunting): authenticate will time out at 1989.608477s 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a149a700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f039c06f150 0x7f039c11aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35638/0 (socket says 172.21.15.112:35638) 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a149a700 1 -- 172.21.15.112:0/3584851979 learned_addr learned my addr 172.21.15.112:0/3584851979 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a149a700 1 -- 172.21.15.112:0/3584851979 >> v1:172.21.15.112:6790/0 conn(0x7f039c06f150 legacy=0x7f039c11aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 client.?.objecter ms_handle_connect 0x7f039c06f150 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 client.?.objecter ms_handle_connect 0x7f039c06f920 2024-09-18T12:03:37.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a1c9b700 1 -- 172.21.15.112:0/3584851979 >> v1:172.21.15.112:6789/0 conn(0x7f039c06f920 legacy=0x7f039c146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03a0c99700 1 -- 172.21.15.112:0/3584851979 >> v1:172.21.15.129:6789/0 conn(0x7f039c06f520 legacy=0x7f039c11b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086612275 0 0) 0x7f039c11b8b0 con 0x7f039c06f150 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): my global_id is 154226 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0380004f10 con 0x7f039c06f150 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 client.?.objecter ms_handle_connect 0x7f039c06f520 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627859046 0 0) 0x7f039c097870 con 0x7f039c06f920 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:37.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): my global_id is 164127 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f039c11b8b0 con 0x7f039c06f920 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359444219 0 0) 0x7f039c09abc0 con 0x7f039c06f520 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): my global_id is 144313 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f039c097870 con 0x7f039c06f520 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697979513 0 0) 0x7f039c11b8b0 con 0x7f039c06f920 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f039c09abc0 con 0x7f039c06f920 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007674111 0 0) 0x7f0380004f10 con 0x7f039c06f150 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f039c11b8b0 con 0x7f039c06f150 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0394004bb0 con 0x7f039c06f920 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:37.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): dump: 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:37.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f03980049d0 con 0x7f039c06f150 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.765+0000 7f03917fa700 10 monclient(hunting): dump: 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:37.781 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1024374266 0 0) 0x7f039c09abc0 con 0x7f039c06f920 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 >> v1:172.21.15.112:6790/0 conn(0x7f039c06f150 legacy=0x7f039c11aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 >> v1:172.21.15.129:6789/0 conn(0x7f039c06f520 legacy=0x7f039c11b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 10 monclient: _finish_hunting 0 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 monclient: found mon.a 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c13b240 con 0x7f039c06f920 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 10 monclient: _finish_auth 0 2024-09-18T12:03:37.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 5 monclient: authenticate success, global_id 164127 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 10 monclient: _renew_subs 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c11e0e0 con 0x7f039c06f920 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 10 client.164127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 10 monclient: _renew_subs 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c1395d0 con 0x7f039c06f920 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 1 librados: init done 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0394001ce0 con 0x7f039c06f920 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f03940051a0 con 0x7f039c06f920 2024-09-18T12:03:37.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f0392ffd700 4 set_mon_vals no callback set 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 10 monclient: dump: 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:37.784 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037c0050b0 con 0x7f039c06f920 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f039400fa20 con 0x7f039c06f920 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f0394043dd0 con 0x7f039c06f920 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 10 client.164127.objecter ms_dispatch 0x7f039c000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 3 client.164127.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 3 client.164127.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03917fa700 20 client.164127.objecter dump_active .. 0 homeless 2024-09-18T12:03:37.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.769+0000 7f03a149a700 10 monclient: get_auth_request con 0x7f0380038240 auth_method 0 2024-09-18T12:03:37.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.773+0000 7f03917fa700 10 client.164127.objecter ms_handle_connect 0x7f0380038240 2024-09-18T12:03:37.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.777+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f03940148f0 con 0x7f039c06f920 2024-09-18T12:03:37.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.777+0000 7f03917fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:37.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.777+0000 7f03917fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.997+0000 7f03a249c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:03:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.997+0000 7f03a249c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:03:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.997+0000 7f03a249c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:37.997+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f037c0058e0 con 0x7f039c06f920 2024-09-18T12:03:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03917fa700 1 -- 172.21.15.112:0/3584851979 <== mon.0 v1:172.21.15.112:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3978708402) 0x7f0394014710 con 0x7f039c06f920 2024-09-18T12:03:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03917fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:03:38.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03917fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:03:38.004 INFO:teuthology.orchestra.run.smithi112.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","modified":"2024-09-18T11:43:39.500070Z","created":"2024-09-18T11:43:39.500070Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6790","nonce":0}]},"addr":"172.21.15.112:6790/0","public_addr":"172.21.15.112:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 10 librados: watch_flush enter 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 10 librados: watch_flush exit 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 20 client.164127.objecter shutdown clearing up homeless session... 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 10 client.164127.objecter successfully canceled tick 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 >> v1:172.21.15.129:6812/20806 conn(0x7f0380038240 legacy=0x7f038003a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 10 monclient: shutdown 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 >> v1:172.21.15.112:6789/0 conn(0x7f039c06f920 legacy=0x7f039c146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 shutdown_connections 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 >> 172.21.15.112:0/3584851979 conn(0x7f039c0a1e30 msgr2=0x7f039c09a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 shutdown_connections 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 1 -- 172.21.15.112:0/3584851979 wait complete. 2024-09-18T12:03:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.001+0000 7f03a249c700 1 librados: shutdown 2024-09-18T12:03:38.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:03:38.017 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T12:03:38.026 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-18T12:03:38.026 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:03:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.129+0000 7fb6685f7700 10 monclient: get_monmap_and_config 2024-09-18T12:03:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.129+0000 7fb6685f7700 10 monclient: build_initial_monmap 2024-09-18T12:03:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.129+0000 7fb6685f7700 10 monclient: monmap: 2024-09-18T12:03:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:38.136208+0000 2024-09-18T12:03:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:38.136208+0000 2024-09-18T12:03:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:38.136 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 1 Processor -- start 2024-09-18T12:03:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 1 -- start start 2024-09-18T12:03:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 10 monclient: init 2024-09-18T12:03:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:03:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 10 monclient(hunting): picked mon.noname-a con 0x7fb6600a2200 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 10 monclient(hunting): picked mon.noname-b con 0x7fb6600a01e0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 10 monclient(hunting): picked mon.noname-c con 0x7fb660138600 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:38.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66006e890 con 0x7fb6600a2200 2024-09-18T12:03:38.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6600a16e0 con 0x7fb660138600 2024-09-18T12:03:38.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6600a25d0 con 0x7fb6600a01e0 2024-09-18T12:03:38.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:38.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb6685f7700 10 monclient(hunting): authenticate will time out at 1989.976982s 2024-09-18T12:03:38.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb666393700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb6600a2200 0x7fb66009ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51960/0 (socket says 172.21.15.112:51960) 2024-09-18T12:03:38.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb666393700 1 -- 172.21.15.112:0/3329404399 learned_addr learned my addr 172.21.15.112:0/3329404399 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:38.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb666393700 1 -- 172.21.15.112:0/3329404399 >> v1:172.21.15.112:6789/0 conn(0x7fb6600a2200 legacy=0x7fb66009ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb665b92700 1 -- 172.21.15.112:0/3329404399 >> v1:172.21.15.112:6790/0 conn(0x7fb6600a01e0 legacy=0x7fb66009b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.133+0000 7fb666b94700 1 -- 172.21.15.112:0/3329404399 >> v1:172.21.15.129:6789/0 conn(0x7fb660138600 legacy=0x7fb66013a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601498235 0 0) 0x7fb650001450 con 0x7fb6600a2200 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): my global_id is 164130 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64c003570 con 0x7fb6600a2200 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970869432 0 0) 0x7fb6600a16e0 con 0x7fb660138600 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): my global_id is 144316 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb650001450 con 0x7fb660138600 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259249522 0 0) 0x7fb6600a25d0 con 0x7fb6600a01e0 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient(hunting): my global_id is 154229 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6600a16e0 con 0x7fb6600a01e0 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682185244 0 0) 0x7fb6600a16e0 con 0x7fb6600a01e0 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 >> v1:172.21.15.129:6789/0 conn(0x7fb660138600 legacy=0x7fb66013a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 >> v1:172.21.15.112:6789/0 conn(0x7fb6600a2200 legacy=0x7fb66009ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: _finish_hunting 0 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 monclient: found mon.noname-b 2024-09-18T12:03:38.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66009bc90 con 0x7fb6600a01e0 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: _finish_auth 0 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 5 monclient: authenticate success, global_id 154229 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb65c0018f0 con 0x7fb6600a01e0 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: dump: 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb65c001a50 con 0x7fb6600a01e0 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:38.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 1 -- 172.21.15.112:0/3329404399 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb65c001f00 con 0x7fb6600a01e0 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 4 set_mon_vals no callback set 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665391700 10 monclient: dump: 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.145 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient: shutdown 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- 172.21.15.112:0/3329404399 >> v1:172.21.15.112:6790/0 conn(0x7fb6600a01e0 legacy=0x7fb66009b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- 172.21.15.112:0/3329404399 shutdown_connections 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- 172.21.15.112:0/3329404399 >> 172.21.15.112:0/3329404399 conn(0x7fb6600680d0 msgr2=0x7fb6600a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- 172.21.15.112:0/3329404399 shutdown_connections 2024-09-18T12:03:38.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- 172.21.15.112:0/3329404399 wait complete. 2024-09-18T12:03:38.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient: build_initial_monmap 2024-09-18T12:03:38.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient: monmap: 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 librados: starting msgr at 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 librados: starting objecter 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 Processor -- start 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- start start 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 librados: setting wanted keys 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 librados: calling monclient init 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient: init 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient(hunting): picked mon.noname-b con 0x7fb6600a01e0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient(hunting): picked mon.noname-a con 0x7fb6600a2200 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient(hunting): picked mon.noname-c con 0x7fb660138600 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6600a0f80 con 0x7fb6600a2200 2024-09-18T12:03:38.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6601188f0 con 0x7fb6600a01e0 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66009ec70 con 0x7fb660138600 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb6685f7700 10 monclient(hunting): authenticate will time out at 1989.983174s 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb666b94700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb660138600 0x7fb6601498e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35656/0 (socket says 172.21.15.112:35656) 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb665b92700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb6600a2200 0x7fb6601314c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51964/0 (socket says 172.21.15.112:51964) 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.137+0000 7fb666b94700 1 -- 172.21.15.112:0/3391120182 learned_addr learned my addr 172.21.15.112:0/3391120182 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb666b94700 1 -- 172.21.15.112:0/3391120182 >> v1:172.21.15.112:6790/0 conn(0x7fb660138600 legacy=0x7fb6601498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb665b92700 1 -- 172.21.15.112:0/3391120182 >> v1:172.21.15.112:6789/0 conn(0x7fb6600a2200 legacy=0x7fb6601314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 client.?.objecter ms_handle_connect 0x7fb660138600 2024-09-18T12:03:38.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 client.?.objecter ms_handle_connect 0x7fb6600a2200 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 client.?.objecter ms_handle_connect 0x7fb6600a01e0 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb666393700 1 -- 172.21.15.112:0/3391120182 >> v1:172.21.15.129:6789/0 conn(0x7fb6600a01e0 legacy=0x7fb660141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022562479 0 0) 0x7fb6600a0f80 con 0x7fb6600a2200 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): my global_id is 164133 2024-09-18T12:03:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb644004f10 con 0x7fb6600a2200 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307242933 0 0) 0x7fb66009ec70 con 0x7fb660138600 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): my global_id is 154232 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6600a0f80 con 0x7fb660138600 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584332406 0 0) 0x7fb6601188f0 con 0x7fb6600a01e0 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): my global_id is 144319 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb66009ec70 con 0x7fb6600a01e0 2024-09-18T12:03:38.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4092949750 0 0) 0x7fb6600a0f80 con 0x7fb660138600 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6601188f0 con 0x7fb660138600 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb658002280 con 0x7fb660138600 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): dump: 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.157 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 791700161 0 0) 0x7fb644004f10 con 0x7fb6600a2200 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6600a0f80 con 0x7fb6600a2200 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb65c001bc0 con 0x7fb6600a2200 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient(hunting): dump: 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4274533845 0 0) 0x7fb6601188f0 con 0x7fb660138600 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 >> v1:172.21.15.129:6789/0 conn(0x7fb6600a01e0 legacy=0x7fb660141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 >> v1:172.21.15.112:6789/0 conn(0x7fb6600a2200 legacy=0x7fb6601314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient: _finish_hunting 0 2024-09-18T12:03:38.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 monclient: found mon.c 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb660131bd0 con 0x7fb660138600 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient: _finish_auth 0 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 5 monclient: authenticate success, global_id 154232 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 10 monclient: _renew_subs 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb660136f40 con 0x7fb660138600 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 10 client.154232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 10 monclient: _renew_subs 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb660137c70 con 0x7fb660138600 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6685f7700 1 librados: init done 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb658002690 con 0x7fb660138600 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb6580029e0 con 0x7fb660138600 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb664b90700 4 set_mon_vals no callback set 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.141+0000 7fb6567fc700 10 monclient: dump: 2024-09-18T12:03:38.161 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6685f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6685f7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6685f7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6480050b0 con 0x7fb660138600 2024-09-18T12:03:38.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb658002c40 con 0x7fb660138600 2024-09-18T12:03:38.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fb658043ef0 con 0x7fb660138600 2024-09-18T12:03:38.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6567fc700 10 client.154232.objecter ms_dispatch 0x7fb660000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:38.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6567fc700 3 client.154232.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:38.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6567fc700 3 client.154232.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:38.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb6567fc700 20 client.154232.objecter dump_active .. 0 homeless 2024-09-18T12:03:38.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.145+0000 7fb666393700 10 monclient: get_auth_request con 0x7fb6440386e0 auth_method 0 2024-09-18T12:03:38.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.149+0000 7fb6567fc700 10 client.154232.objecter ms_handle_connect 0x7fb6440386e0 2024-09-18T12:03:38.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.153+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fb658014db0 con 0x7fb660138600 2024-09-18T12:03:38.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.153+0000 7fb6567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.153+0000 7fb6567fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:38.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.309+0000 7fb6685f7700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:03:38.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.309+0000 7fb6685f7700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:03:38.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.309+0000 7fb6685f7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.309+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb648005360 con 0x7fb660138600 2024-09-18T12:03:38.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.309+0000 7fb6567fc700 1 -- 172.21.15.112:0/3391120182 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v699) v1 ==== 74+0+12741 (unknown 3608138097 0 2795684330) 0x7fb6580147b0 con 0x7fb660138600 2024-09-18T12:03:38.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.313+0000 7fb6567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:03:38.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.313+0000 7fb6567fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:03:38.315 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T12:03:38.315 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":699,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","created":"2024-09-18T11:44:09.317663+0000","modified":"2024-09-18T12:02:11.898642+0000","last_up_change":"2024-09-18T11:44:19.034653+0000","last_in_change":"2024-09-18T11:44:11.615001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":104,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:44:19.901214+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-18T11:44:20.029398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-18T11:45:24.012859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"ecb0abea-5208-453b-86e2-58b64b0634ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":20984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":20984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":20984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":20984}]},"public_addr":"172.21.15.112:6804/20984","cluster_addr":"172.21.15.112:6805/20984","heartbeat_back_addr":"172.21.15.112:6807/20984","heartbeat_front_addr":"172.21.15.112:6806/20984","state":["exists","up"]},{"osd":1,"uuid":"3164789a-58af-4cf7-bd86-260d70dd02c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":20985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":20985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":20985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":20985}]},"public_addr":"172.21.15.112:6800/20985","cluster_addr":"172.21.15.112:6801/20985","heartbeat_back_addr":"172.21.15.112:6803/20985","heartbeat_front_addr":"172.21.15.112:6802/20985","state":["exists","up"]},{"osd":2,"uuid":"0223fabc-71f3-4642-8818-d99dbd5ba501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":20986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":20986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":20986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":20986}]},"public_addr":"172.21.15.112:6808/20986","cluster_addr":"172.21.15.112:6809/20986","heartbeat_back_addr":"172.21.15.112:6811/20986","heartbeat_front_addr":"172.21.15.112:6810/20986","state":["exists","up"]},{"osd":3,"uuid":"07461389-250c-4e30-8890-4267bddeac4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6804","nonce":21049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6805","nonce":21049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6807","nonce":21049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6806","nonce":21049}]},"public_addr":"172.21.15.129:6804/21049","cluster_addr":"172.21.15.129:6805/21049","heartbeat_back_addr":"172.21.15.129:6807/21049","heartbeat_front_addr":"172.21.15.129:6806/21049","state":["exists","up"]},{"osd":4,"uuid":"ad3a24d7-0278-4137-a499-e7b7337e1698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6808","nonce":21047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6809","nonce":21047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6811","nonce":21047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6810","nonce":21047}]},"public_addr":"172.21.15.129:6808/21047","cluster_addr":"172.21.15.129:6809/21047","heartbeat_back_addr":"172.21.15.129:6811/21047","heartbeat_front_addr":"172.21.15.129:6810/21047","state":["exists","up"]},{"osd":5,"uuid":"174b7f7f-23bc-45c9-9245-7c3edc4b6235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6800","nonce":21048}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6801","nonce":21048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6803","nonce":21048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6802","nonce":21048}]},"public_addr":"172.21.15.129:6800/21048","cluster_addr":"172.21.15.129:6801/21048","heartbeat_back_addr":"172.21.15.129:6803/21048","heartbeat_front_addr":"172.21.15.129:6802/21048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.965457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.813648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.196234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.055175+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.022483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.092059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":13,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":54,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":74,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":82,"snaps":[{"begin":2,"length":1}]},{"pool":90,"snaps":[{"begin":2,"length":1},{"begin":6,"length":2}]},{"pool":92,"snaps":[{"begin":4,"length":4}]},{"pool":103,"snaps":[{"begin":2,"length":1}]},{"pool":109,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":119,"snaps":[{"begin":12,"length":1}]},{"pool":125,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":144,"snaps":[{"begin":14,"length":1}]},{"pool":155,"snaps":[{"begin":17,"length":1}]},{"pool":161,"snaps":[{"begin":19,"length":2}]},{"pool":176,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":5,"length":1}]},{"pool":244,"snaps":[{"begin":7,"length":1}]},{"pool":248,"snaps":[{"begin":10,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:03:38.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.313+0000 7fb6685f7700 10 librados: watch_flush enter 2024-09-18T12:03:38.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 10 librados: watch_flush exit 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 20 client.154232.objecter shutdown clearing up homeless session... 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 10 client.154232.objecter successfully canceled tick 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 >> v1:172.21.15.129:6812/20806 conn(0x7fb6440386e0 legacy=0x7fb64403ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 10 monclient: shutdown 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 >> v1:172.21.15.112:6790/0 conn(0x7fb660138600 legacy=0x7fb6601498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 shutdown_connections 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 >> 172.21.15.112:0/3391120182 conn(0x7fb6600680d0 msgr2=0x7fb66013a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 shutdown_connections 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 1 -- 172.21.15.112:0/3391120182 wait complete. 2024-09-18T12:03:38.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.317+0000 7fb6685f7700 1 librados: shutdown 2024-09-18T12:03:38.336 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T12:03:38.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.429+0000 7fb36dadb700 10 monclient: get_monmap_and_config 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.429+0000 7fb36dadb700 10 monclient: build_initial_monmap 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.429+0000 7fb36dadb700 10 monclient: monmap: 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:38.436099+0000 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:38.436099+0000 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.435 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:38.436 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:38.436 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:38.436 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.436 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 1 Processor -- start 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 1 -- start start 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 10 monclient: init 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 10 monclient(hunting): picked mon.noname-c con 0x7fb3680a2200 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 10 monclient(hunting): picked mon.noname-b con 0x7fb3680a01e0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 10 monclient(hunting): picked mon.noname-a con 0x7fb368138600 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36806e890 con 0x7fb368138600 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680a16e0 con 0x7fb3680a2200 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680a25d0 con 0x7fb3680a01e0 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:38.437 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb36dadb700 10 monclient(hunting): authenticate will time out at 1990.276980s 2024-09-18T12:03:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb35effd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fb3680a01e0 0x7fb36809b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35670/0 (socket says 172.21.15.112:35670) 2024-09-18T12:03:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb35effd700 1 -- 172.21.15.112:0/3475022892 learned_addr learned my addr 172.21.15.112:0/3475022892 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:38.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb35effd700 1 -- 172.21.15.112:0/3475022892 >> v1:172.21.15.112:6790/0 conn(0x7fb3680a01e0 legacy=0x7fb36809b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb367fff700 1 -- 172.21.15.112:0/3475022892 >> v1:172.21.15.112:6789/0 conn(0x7fb368138600 legacy=0x7fb36813a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb3677fe700 1 -- 172.21.15.112:0/3475022892 >> v1:172.21.15.129:6789/0 conn(0x7fb3680a2200 legacy=0x7fb36809ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920974292 0 0) 0x7fb3680a25d0 con 0x7fb3680a01e0 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): my global_id is 154235 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb348003570 con 0x7fb3680a01e0 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458457882 0 0) 0x7fb360001820 con 0x7fb368138600 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): my global_id is 164136 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3680a25d0 con 0x7fb368138600 2024-09-18T12:03:38.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170990497 0 0) 0x7fb3680a16e0 con 0x7fb3680a2200 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 10 monclient(hunting): my global_id is 144322 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.433+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb360001820 con 0x7fb3680a2200 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3575884455 0 0) 0x7fb348003570 con 0x7fb3680a01e0 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 >> v1:172.21.15.129:6789/0 conn(0x7fb3680a2200 legacy=0x7fb36809ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 >> v1:172.21.15.112:6789/0 conn(0x7fb368138600 legacy=0x7fb36813a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 monclient: found mon.noname-b 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36809bc90 con 0x7fb3680a01e0 2024-09-18T12:03:38.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: _finish_auth 0 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 5 monclient: authenticate success, global_id 154235 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb358001940 con 0x7fb3680a01e0 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: dump: 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.443 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265293082 0 0) 0x7fb360001820 con 0x7fb3680a2200 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb350001930 con 0x7fb3680a2200 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb358001bd0 con 0x7fb3680a01e0 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 1 -- 172.21.15.112:0/3475022892 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb358004a90 con 0x7fb3680a01e0 2024-09-18T12:03:38.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 4 set_mon_vals no callback set 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb366ffd700 10 monclient: dump: 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.445 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient: shutdown 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- 172.21.15.112:0/3475022892 >> v1:172.21.15.112:6790/0 conn(0x7fb3680a01e0 legacy=0x7fb36809b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- 172.21.15.112:0/3475022892 shutdown_connections 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- 172.21.15.112:0/3475022892 >> 172.21.15.112:0/3475022892 conn(0x7fb3680680d0 msgr2=0x7fb3680a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- 172.21.15.112:0/3475022892 shutdown_connections 2024-09-18T12:03:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- 172.21.15.112:0/3475022892 wait complete. 2024-09-18T12:03:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient: build_initial_monmap 2024-09-18T12:03:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient: monmap: 2024-09-18T12:03:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 librados: starting msgr at 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 librados: starting objecter 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 Processor -- start 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- start start 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 librados: setting wanted keys 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 librados: calling monclient init 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient: init 2024-09-18T12:03:38.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient(hunting): picked mon.noname-b con 0x7fb3680a01e0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient(hunting): picked mon.noname-c con 0x7fb3680a2200 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient(hunting): picked mon.noname-a con 0x7fb368138600 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb368137710 con 0x7fb368138600 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36809ec50 con 0x7fb3680a01e0 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680a0f80 con 0x7fb3680a2200 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb36dadb700 10 monclient(hunting): authenticate will time out at 1990.283546s 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb367fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb368138600 0x7fb368149730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51976/0 (socket says 172.21.15.112:51976) 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.437+0000 7fb367fff700 1 -- 172.21.15.112:0/3978748148 learned_addr learned my addr 172.21.15.112:0/3978748148 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb367fff700 1 -- 172.21.15.112:0/3978748148 >> v1:172.21.15.112:6789/0 conn(0x7fb368138600 legacy=0x7fb368149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 client.?.objecter ms_handle_connect 0x7fb368138600 2024-09-18T12:03:38.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 client.?.objecter ms_handle_connect 0x7fb3680a2200 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35effd700 1 -- 172.21.15.112:0/3978748148 >> v1:172.21.15.112:6790/0 conn(0x7fb3680a2200 legacy=0x7fb368135920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801073436 0 0) 0x7fb368137710 con 0x7fb368138600 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb3677fe700 1 -- 172.21.15.112:0/3978748148 >> v1:172.21.15.129:6789/0 conn(0x7fb3680a01e0 legacy=0x7fb368135210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): my global_id is 164139 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb34c004f10 con 0x7fb368138600 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 client.?.objecter ms_handle_connect 0x7fb3680a01e0 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034341871 0 0) 0x7fb3680a0f80 con 0x7fb3680a2200 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.454 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): my global_id is 154238 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb368137710 con 0x7fb3680a2200 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126584174 0 0) 0x7fb36809ec50 con 0x7fb3680a01e0 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): my global_id is 144325 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3680a0f80 con 0x7fb3680a01e0 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1735711189 0 0) 0x7fb34c004f10 con 0x7fb368138600 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb36809ec50 con 0x7fb368138600 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb360002280 con 0x7fb368138600 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): dump: 2024-09-18T12:03:38.455 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1413025696 0 0) 0x7fb368137710 con 0x7fb3680a2200 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb34c004f10 con 0x7fb3680a2200 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb358001360 con 0x7fb3680a2200 2024-09-18T12:03:38.456 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): dump: 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324596974 0 0) 0x7fb3680a0f80 con 0x7fb3680a01e0 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb368137710 con 0x7fb3680a01e0 2024-09-18T12:03:38.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb350001980 con 0x7fb3680a01e0 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient(hunting): dump: 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1384415338 0 0) 0x7fb34c004f10 con 0x7fb3680a2200 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 >> v1:172.21.15.129:6789/0 conn(0x7fb3680a01e0 legacy=0x7fb368135210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 >> v1:172.21.15.112:6789/0 conn(0x7fb368138600 legacy=0x7fb368149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient: _finish_hunting 0 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 monclient: found mon.c 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb368136030 con 0x7fb3680a2200 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient: _finish_auth 0 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:38.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 5 monclient: authenticate success, global_id 154238 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 10 monclient: _renew_subs 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb368137220 con 0x7fb3680a2200 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 10 client.154238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 10 monclient: _renew_subs 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb358004d80 con 0x7fb3680a2200 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb368136d10 con 0x7fb3680a2200 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb36dadb700 1 librados: init done 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fb3580050d0 con 0x7fb3680a2200 2024-09-18T12:03:38.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb35ffff700 10 monclient: dump: 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.441+0000 7fb3667fc700 4 set_mon_vals no callback set 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb36dadb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb36dadb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb36dadb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb368118a70 con 0x7fb3680a2200 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fb35800f950 con 0x7fb3680a2200 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fb358043b20 con 0x7fb3680a2200 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb35ffff700 10 client.154238.objecter ms_dispatch 0x7fb368000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb35ffff700 3 client.154238.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb35ffff700 3 client.154238.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb35ffff700 20 client.154238.objecter dump_active .. 0 homeless 2024-09-18T12:03:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.445+0000 7fb3677fe700 10 monclient: get_auth_request con 0x7fb34c038540 auth_method 0 2024-09-18T12:03:38.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.449+0000 7fb35ffff700 10 client.154238.objecter ms_handle_connect 0x7fb34c038540 2024-09-18T12:03:38.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.453+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fb358001580 con 0x7fb3680a2200 2024-09-18T12:03:38.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.453+0000 7fb35ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.463 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.453+0000 7fb35ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:38.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.613+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb36805f2e0 con 0x7fb34c038540 2024-09-18T12:03:38.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb35ffff700 1 -- 172.21.15.112:0/3978748148 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134033 (unknown 2965378022 0 767431492) 0x7fb36805f2e0 con 0x7fb34c038540 2024-09-18T12:03:38.629 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T12:03:38.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 10 librados: watch_flush enter 2024-09-18T12:03:38.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 10 librados: watch_flush exit 2024-09-18T12:03:38.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 20 client.154238.objecter shutdown clearing up homeless session... 2024-09-18T12:03:38.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 10 client.154238.objecter successfully canceled tick 2024-09-18T12:03:38.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 >> v1:172.21.15.129:6812/20806 conn(0x7fb34c038540 legacy=0x7fb34c03a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 10 monclient: shutdown 2024-09-18T12:03:38.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:38.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 >> v1:172.21.15.112:6790/0 conn(0x7fb3680a2200 legacy=0x7fb368135920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 shutdown_connections 2024-09-18T12:03:38.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.625+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 >> 172.21.15.112:0/3978748148 conn(0x7fb3680680d0 msgr2=0x7fb36813a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:38.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.629+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 shutdown_connections 2024-09-18T12:03:38.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.629+0000 7fb36dadb700 1 -- 172.21.15.112:0/3978748148 wait complete. 2024-09-18T12:03:38.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.629+0000 7fb36dadb700 1 librados: shutdown 2024-09-18T12:03:38.632 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T12:03:38.645 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1274,"stamp":"2024-09-18T12:03:38.380870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1898908,"kb_used_data":56444,"kb_used_omap":9,"kb_used_meta":1842294,"kb_avail":564332132,"statfs":{"total":579820584960,"available":577876103168,"internally_reserved":0,"allocated":57798656,"data_stored":30786034,"data_compressed":320902,"data_compressed_allocated":876544,"data_compressed_original":2744320,"omap_allocated":10224,"internal_metadata":1886509072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003849"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522453+0000","last_change":"2024-09-18T11:45:25.374991+0000","last_active":"2024-09-18T12:02:11.522453+0000","last_peered":"2024-09-18T12:02:11.522453+0000","last_clean":"2024-09-18T12:02:11.522453+0000","last_became_active":"2024-09-18T11:45:25.373642+0000","last_became_peered":"2024-09-18T11:45:25.373642+0000","last_unstale":"2024-09-18T12:02:11.522453+0000","last_undegraded":"2024-09-18T12:02:11.522453+0000","last_fullsized":"2024-09-18T12:02:11.522453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:57:36.040393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519515+0000","last_change":"2024-09-18T11:45:25.329845+0000","last_active":"2024-09-18T12:02:11.519515+0000","last_peered":"2024-09-18T12:02:11.519515+0000","last_clean":"2024-09-18T12:02:11.519515+0000","last_became_active":"2024-09-18T11:45:25.328032+0000","last_became_peered":"2024-09-18T11:45:25.328032+0000","last_unstale":"2024-09-18T12:02:11.519515+0000","last_undegraded":"2024-09-18T12:02:11.519515+0000","last_fullsized":"2024-09-18T12:02:11.519515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:13.680321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913257+0000","last_change":"2024-09-18T11:45:25.306434+0000","last_active":"2024-09-18T12:02:10.913257+0000","last_peered":"2024-09-18T12:02:10.913257+0000","last_clean":"2024-09-18T12:02:10.913257+0000","last_became_active":"2024-09-18T11:45:25.305872+0000","last_became_peered":"2024-09-18T11:45:25.305872+0000","last_unstale":"2024-09-18T12:02:10.913257+0000","last_undegraded":"2024-09-18T12:02:10.913257+0000","last_fullsized":"2024-09-18T12:02:10.913257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:30:00.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519584+0000","last_change":"2024-09-18T11:45:25.411469+0000","last_active":"2024-09-18T12:02:11.519584+0000","last_peered":"2024-09-18T12:02:11.519584+0000","last_clean":"2024-09-18T12:02:11.519584+0000","last_became_active":"2024-09-18T11:45:25.411133+0000","last_became_peered":"2024-09-18T11:45:25.411133+0000","last_unstale":"2024-09-18T12:02:11.519584+0000","last_undegraded":"2024-09-18T12:02:11.519584+0000","last_fullsized":"2024-09-18T12:02:11.519584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:02:10.494289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523123+0000","last_change":"2024-09-18T11:45:25.400321+0000","last_active":"2024-09-18T12:02:11.523123+0000","last_peered":"2024-09-18T12:02:11.523123+0000","last_clean":"2024-09-18T12:02:11.523123+0000","last_became_active":"2024-09-18T11:45:25.399898+0000","last_became_peered":"2024-09-18T11:45:25.399898+0000","last_unstale":"2024-09-18T12:02:11.523123+0000","last_undegraded":"2024-09-18T12:02:11.523123+0000","last_fullsized":"2024-09-18T12:02:11.523123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:14.033638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.520050+0000","last_change":"2024-09-18T11:45:25.397202+0000","last_active":"2024-09-18T12:02:11.520050+0000","last_peered":"2024-09-18T12:02:11.520050+0000","last_clean":"2024-09-18T12:02:11.520050+0000","last_became_active":"2024-09-18T11:45:25.396975+0000","last_became_peered":"2024-09-18T11:45:25.396975+0000","last_unstale":"2024-09-18T12:02:11.520050+0000","last_undegraded":"2024-09-18T12:02:11.520050+0000","last_fullsized":"2024-09-18T12:02:11.520050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:37:37.634734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.910998+0000","last_change":"2024-09-18T11:45:25.341093+0000","last_active":"2024-09-18T12:02:10.910998+0000","last_peered":"2024-09-18T12:02:10.910998+0000","last_clean":"2024-09-18T12:02:10.910998+0000","last_became_active":"2024-09-18T11:45:25.340726+0000","last_became_peered":"2024-09-18T11:45:25.340726+0000","last_unstale":"2024-09-18T12:02:10.910998+0000","last_undegraded":"2024-09-18T12:02:10.910998+0000","last_fullsized":"2024-09-18T12:02:10.910998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:36:59.172751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960537+0000","last_change":"2024-09-18T11:45:25.408737+0000","last_active":"2024-09-18T12:02:10.960537+0000","last_peered":"2024-09-18T12:02:10.960537+0000","last_clean":"2024-09-18T12:02:10.960537+0000","last_became_active":"2024-09-18T11:45:25.408464+0000","last_became_peered":"2024-09-18T11:45:25.408464+0000","last_unstale":"2024-09-18T12:02:10.960537+0000","last_undegraded":"2024-09-18T12:02:10.960537+0000","last_fullsized":"2024-09-18T12:02:10.960537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:44:57.076431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.909867+0000","last_change":"2024-09-18T11:45:25.344531+0000","last_active":"2024-09-18T12:02:10.909867+0000","last_peered":"2024-09-18T12:02:10.909867+0000","last_clean":"2024-09-18T12:02:10.909867+0000","last_became_active":"2024-09-18T11:45:25.343097+0000","last_became_peered":"2024-09-18T11:45:25.343097+0000","last_unstale":"2024-09-18T12:02:10.909867+0000","last_undegraded":"2024-09-18T12:02:10.909867+0000","last_fullsized":"2024-09-18T12:02:10.909867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:49.438992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519649+0000","last_change":"2024-09-18T11:45:25.377493+0000","last_active":"2024-09-18T12:02:11.519649+0000","last_peered":"2024-09-18T12:02:11.519649+0000","last_clean":"2024-09-18T12:02:11.519649+0000","last_became_active":"2024-09-18T11:45:25.377194+0000","last_became_peered":"2024-09-18T11:45:25.377194+0000","last_unstale":"2024-09-18T12:02:11.519649+0000","last_undegraded":"2024-09-18T12:02:11.519649+0000","last_fullsized":"2024-09-18T12:02:11.519649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:44:53.107224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522770+0000","last_change":"2024-09-18T11:45:25.346404+0000","last_active":"2024-09-18T12:02:11.522770+0000","last_peered":"2024-09-18T12:02:11.522770+0000","last_clean":"2024-09-18T12:02:11.522770+0000","last_became_active":"2024-09-18T11:45:25.341682+0000","last_became_peered":"2024-09-18T11:45:25.341682+0000","last_unstale":"2024-09-18T12:02:11.522770+0000","last_undegraded":"2024-09-18T12:02:11.522770+0000","last_fullsized":"2024-09-18T12:02:11.522770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:04.010096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522588+0000","last_change":"2024-09-18T11:45:25.383080+0000","last_active":"2024-09-18T12:02:11.522588+0000","last_peered":"2024-09-18T12:02:11.522588+0000","last_clean":"2024-09-18T12:02:11.522588+0000","last_became_active":"2024-09-18T11:45:25.377663+0000","last_became_peered":"2024-09-18T11:45:25.377663+0000","last_unstale":"2024-09-18T12:02:11.522588+0000","last_undegraded":"2024-09-18T12:02:11.522588+0000","last_fullsized":"2024-09-18T12:02:11.522588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:46:12.443901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":649,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.914509+0000","last_change":"2024-09-18T11:44:24.299141+0000","last_active":"2024-09-18T12:02:10.914509+0000","last_peered":"2024-09-18T12:02:10.914509+0000","last_clean":"2024-09-18T12:02:10.914509+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T12:02:10.914509+0000","last_undegraded":"2024-09-18T12:02:10.914509+0000","last_fullsized":"2024-09-18T12:02:10.914509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:35:52.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960533+0000","last_change":"2024-09-18T11:45:25.405730+0000","last_active":"2024-09-18T12:02:10.960533+0000","last_peered":"2024-09-18T12:02:10.960533+0000","last_clean":"2024-09-18T12:02:10.960533+0000","last_became_active":"2024-09-18T11:45:25.405515+0000","last_became_peered":"2024-09-18T11:45:25.405515+0000","last_unstale":"2024-09-18T12:02:10.960533+0000","last_undegraded":"2024-09-18T12:02:10.960533+0000","last_fullsized":"2024-09-18T12:02:10.960533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:39:09.362580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911684+0000","last_change":"2024-09-18T11:44:24.208317+0000","last_active":"2024-09-18T12:02:10.911684+0000","last_peered":"2024-09-18T12:02:10.911684+0000","last_clean":"2024-09-18T12:02:10.911684+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T12:02:10.911684+0000","last_undegraded":"2024-09-18T12:02:10.911684+0000","last_fullsized":"2024-09-18T12:02:10.911684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:54.979951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913034+0000","last_change":"2024-09-18T11:45:25.306008+0000","last_active":"2024-09-18T12:02:10.913034+0000","last_peered":"2024-09-18T12:02:10.913034+0000","last_clean":"2024-09-18T12:02:10.913034+0000","last_became_active":"2024-09-18T11:45:25.305686+0000","last_became_peered":"2024-09-18T11:45:25.305686+0000","last_unstale":"2024-09-18T12:02:10.913034+0000","last_undegraded":"2024-09-18T12:02:10.913034+0000","last_fullsized":"2024-09-18T12:02:10.913034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:31:41.349569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519991+0000","last_change":"2024-09-18T11:44:24.213224+0000","last_active":"2024-09-18T12:02:11.519991+0000","last_peered":"2024-09-18T12:02:11.519991+0000","last_clean":"2024-09-18T12:02:11.519991+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T12:02:11.519991+0000","last_undegraded":"2024-09-18T12:02:11.519991+0000","last_fullsized":"2024-09-18T12:02:11.519991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:47.310939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"20'7","reported_seq":660,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.910044+0000","last_change":"2024-09-18T11:45:25.403798+0000","last_active":"2024-09-18T12:02:10.910044+0000","last_peered":"2024-09-18T12:02:10.910044+0000","last_clean":"2024-09-18T12:02:10.910044+0000","last_became_active":"2024-09-18T11:45:25.403586+0000","last_became_peered":"2024-09-18T11:45:25.403586+0000","last_unstale":"2024-09-18T12:02:10.910044+0000","last_undegraded":"2024-09-18T12:02:10.910044+0000","last_fullsized":"2024-09-18T12:02:10.910044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:53:41.121574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":654,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522879+0000","last_change":"2024-09-18T11:44:24.507933+0000","last_active":"2024-09-18T12:02:11.522879+0000","last_peered":"2024-09-18T12:02:11.522879+0000","last_clean":"2024-09-18T12:02:11.522879+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T12:02:11.522879+0000","last_undegraded":"2024-09-18T12:02:11.522879+0000","last_fullsized":"2024-09-18T12:02:11.522879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:56.697013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522889+0000","last_change":"2024-09-18T11:45:25.369556+0000","last_active":"2024-09-18T12:02:11.522889+0000","last_peered":"2024-09-18T12:02:11.522889+0000","last_clean":"2024-09-18T12:02:11.522889+0000","last_became_active":"2024-09-18T11:45:25.367362+0000","last_became_peered":"2024-09-18T11:45:25.367362+0000","last_unstale":"2024-09-18T12:02:11.522889+0000","last_undegraded":"2024-09-18T12:02:11.522889+0000","last_fullsized":"2024-09-18T12:02:11.522889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:23:17.811113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":743,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.907687+0000","last_change":"2024-09-18T11:44:21.097910+0000","last_active":"2024-09-18T12:02:10.907687+0000","last_peered":"2024-09-18T12:02:10.907687+0000","last_clean":"2024-09-18T12:02:10.907687+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T12:02:10.907687+0000","last_undegraded":"2024-09-18T12:02:10.907687+0000","last_fullsized":"2024-09-18T12:02:10.907687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:48.394073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":655,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522634+0000","last_change":"2024-09-18T11:44:24.508079+0000","last_active":"2024-09-18T12:02:11.522634+0000","last_peered":"2024-09-18T12:02:11.522634+0000","last_clean":"2024-09-18T12:02:11.522634+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T12:02:11.522634+0000","last_undegraded":"2024-09-18T12:02:11.522634+0000","last_fullsized":"2024-09-18T12:02:11.522634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:41:30.648862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913627+0000","last_change":"2024-09-18T11:45:25.407247+0000","last_active":"2024-09-18T12:02:10.913627+0000","last_peered":"2024-09-18T12:02:10.913627+0000","last_clean":"2024-09-18T12:02:10.913627+0000","last_became_active":"2024-09-18T11:45:25.406430+0000","last_became_peered":"2024-09-18T11:45:25.406430+0000","last_unstale":"2024-09-18T12:02:10.913627+0000","last_undegraded":"2024-09-18T12:02:10.913627+0000","last_fullsized":"2024-09-18T12:02:10.913627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:56:03.017626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"17'2","reported_seq":660,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519718+0000","last_change":"2024-09-18T11:44:24.214482+0000","last_active":"2024-09-18T12:02:11.519718+0000","last_peered":"2024-09-18T12:02:11.519718+0000","last_clean":"2024-09-18T12:02:11.519718+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T12:02:11.519718+0000","last_undegraded":"2024-09-18T12:02:11.519718+0000","last_fullsized":"2024-09-18T12:02:11.519718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:26:34.055867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523238+0000","last_change":"2024-09-18T11:45:25.365786+0000","last_active":"2024-09-18T12:02:11.523238+0000","last_peered":"2024-09-18T12:02:11.523238+0000","last_clean":"2024-09-18T12:02:11.523238+0000","last_became_active":"2024-09-18T11:45:25.363235+0000","last_became_peered":"2024-09-18T11:45:25.363235+0000","last_unstale":"2024-09-18T12:02:11.523238+0000","last_undegraded":"2024-09-18T12:02:11.523238+0000","last_fullsized":"2024-09-18T12:02:11.523238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:15:20.066742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":649,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.912589+0000","last_change":"2024-09-18T11:44:24.298975+0000","last_active":"2024-09-18T12:02:10.912589+0000","last_peered":"2024-09-18T12:02:10.912589+0000","last_clean":"2024-09-18T12:02:10.912589+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T12:02:10.912589+0000","last_undegraded":"2024-09-18T12:02:10.912589+0000","last_fullsized":"2024-09-18T12:02:10.912589+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:27.175495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960440+0000","last_change":"2024-09-18T11:45:25.356461+0000","last_active":"2024-09-18T12:02:10.960440+0000","last_peered":"2024-09-18T12:02:10.960440+0000","last_clean":"2024-09-18T12:02:10.960440+0000","last_became_active":"2024-09-18T11:45:25.356178+0000","last_became_peered":"2024-09-18T11:45:25.356178+0000","last_unstale":"2024-09-18T12:02:10.960440+0000","last_undegraded":"2024-09-18T12:02:10.960440+0000","last_fullsized":"2024-09-18T12:02:10.960440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:06:26.698280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.908011+0000","last_change":"2024-09-18T11:44:24.208252+0000","last_active":"2024-09-18T12:02:10.908011+0000","last_peered":"2024-09-18T12:02:10.908011+0000","last_clean":"2024-09-18T12:02:10.908011+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T12:02:10.908011+0000","last_undegraded":"2024-09-18T12:02:10.908011+0000","last_fullsized":"2024-09-18T12:02:10.908011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:08:27.257954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.520056+0000","last_change":"2024-09-18T11:45:25.396966+0000","last_active":"2024-09-18T12:02:11.520056+0000","last_peered":"2024-09-18T12:02:11.520056+0000","last_clean":"2024-09-18T12:02:11.520056+0000","last_became_active":"2024-09-18T11:45:25.396428+0000","last_became_peered":"2024-09-18T11:45:25.396428+0000","last_unstale":"2024-09-18T12:02:11.520056+0000","last_undegraded":"2024-09-18T12:02:11.520056+0000","last_fullsized":"2024-09-18T12:02:11.520056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:08:22.504624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911592+0000","last_change":"2024-09-18T11:45:25.399933+0000","last_active":"2024-09-18T12:02:10.911592+0000","last_peered":"2024-09-18T12:02:10.911592+0000","last_clean":"2024-09-18T12:02:10.911592+0000","last_became_active":"2024-09-18T11:45:25.399695+0000","last_became_peered":"2024-09-18T11:45:25.399695+0000","last_unstale":"2024-09-18T12:02:10.911592+0000","last_undegraded":"2024-09-18T12:02:10.911592+0000","last_fullsized":"2024-09-18T12:02:10.911592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:01:15.698698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913437+0000","last_change":"2024-09-18T11:45:25.387334+0000","last_active":"2024-09-18T12:02:10.913437+0000","last_peered":"2024-09-18T12:02:10.913437+0000","last_clean":"2024-09-18T12:02:10.913437+0000","last_became_active":"2024-09-18T11:45:25.386932+0000","last_became_peered":"2024-09-18T11:45:25.386932+0000","last_unstale":"2024-09-18T12:02:10.913437+0000","last_undegraded":"2024-09-18T12:02:10.913437+0000","last_fullsized":"2024-09-18T12:02:10.913437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:34:23.587779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522768+0000","last_change":"2024-09-18T11:45:25.403356+0000","last_active":"2024-09-18T12:02:11.522768+0000","last_peered":"2024-09-18T12:02:11.522768+0000","last_clean":"2024-09-18T12:02:11.522768+0000","last_became_active":"2024-09-18T11:45:25.401979+0000","last_became_peered":"2024-09-18T11:45:25.401979+0000","last_unstale":"2024-09-18T12:02:11.522768+0000","last_undegraded":"2024-09-18T12:02:11.522768+0000","last_fullsized":"2024-09-18T12:02:11.522768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:29:14.433201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523103+0000","last_change":"2024-09-18T11:45:25.381976+0000","last_active":"2024-09-18T12:02:11.523103+0000","last_peered":"2024-09-18T12:02:11.523103+0000","last_clean":"2024-09-18T12:02:11.523103+0000","last_became_active":"2024-09-18T11:45:25.381652+0000","last_became_peered":"2024-09-18T11:45:25.381652+0000","last_unstale":"2024-09-18T12:02:11.523103+0000","last_undegraded":"2024-09-18T12:02:11.523103+0000","last_fullsized":"2024-09-18T12:02:11.523103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:21.420291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522937+0000","last_change":"2024-09-18T11:45:25.381748+0000","last_active":"2024-09-18T12:02:11.522937+0000","last_peered":"2024-09-18T12:02:11.522937+0000","last_clean":"2024-09-18T12:02:11.522937+0000","last_became_active":"2024-09-18T11:45:25.379657+0000","last_became_peered":"2024-09-18T11:45:25.379657+0000","last_unstale":"2024-09-18T12:02:11.522937+0000","last_undegraded":"2024-09-18T12:02:11.522937+0000","last_fullsized":"2024-09-18T12:02:11.522937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:44:38.923827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523379+0000","last_change":"2024-09-18T11:45:25.340204+0000","last_active":"2024-09-18T12:02:11.523379+0000","last_peered":"2024-09-18T12:02:11.523379+0000","last_clean":"2024-09-18T12:02:11.523379+0000","last_became_active":"2024-09-18T11:45:25.338772+0000","last_became_peered":"2024-09-18T11:45:25.338772+0000","last_unstale":"2024-09-18T12:02:11.523379+0000","last_undegraded":"2024-09-18T12:02:11.523379+0000","last_fullsized":"2024-09-18T12:02:11.523379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:09.409329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522520+0000","last_change":"2024-09-18T11:45:25.369783+0000","last_active":"2024-09-18T12:02:11.522520+0000","last_peered":"2024-09-18T12:02:11.522520+0000","last_clean":"2024-09-18T12:02:11.522520+0000","last_became_active":"2024-09-18T11:45:25.369294+0000","last_became_peered":"2024-09-18T11:45:25.369294+0000","last_unstale":"2024-09-18T12:02:11.522520+0000","last_undegraded":"2024-09-18T12:02:11.522520+0000","last_fullsized":"2024-09-18T12:02:11.522520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:20:30.771482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.912743+0000","last_change":"2024-09-18T11:45:25.393624+0000","last_active":"2024-09-18T12:02:10.912743+0000","last_peered":"2024-09-18T12:02:10.912743+0000","last_clean":"2024-09-18T12:02:10.912743+0000","last_became_active":"2024-09-18T11:45:25.393093+0000","last_became_peered":"2024-09-18T11:45:25.393093+0000","last_unstale":"2024-09-18T12:02:10.912743+0000","last_undegraded":"2024-09-18T12:02:10.912743+0000","last_fullsized":"2024-09-18T12:02:10.912743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:17:30.506236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911484+0000","last_change":"2024-09-18T11:45:25.382303+0000","last_active":"2024-09-18T12:02:10.911484+0000","last_peered":"2024-09-18T12:02:10.911484+0000","last_clean":"2024-09-18T12:02:10.911484+0000","last_became_active":"2024-09-18T11:45:25.380362+0000","last_became_peered":"2024-09-18T11:45:25.380362+0000","last_unstale":"2024-09-18T12:02:10.911484+0000","last_undegraded":"2024-09-18T12:02:10.911484+0000","last_fullsized":"2024-09-18T12:02:10.911484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:11:31.985540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523224+0000","last_change":"2024-09-18T11:45:25.358090+0000","last_active":"2024-09-18T12:02:11.523224+0000","last_peered":"2024-09-18T12:02:11.523224+0000","last_clean":"2024-09-18T12:02:11.523224+0000","last_became_active":"2024-09-18T11:45:25.357477+0000","last_became_peered":"2024-09-18T11:45:25.357477+0000","last_unstale":"2024-09-18T12:02:11.523224+0000","last_undegraded":"2024-09-18T12:02:11.523224+0000","last_fullsized":"2024-09-18T12:02:11.523224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:54:53.792263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960338+0000","last_change":"2024-09-18T11:45:25.394821+0000","last_active":"2024-09-18T12:02:10.960338+0000","last_peered":"2024-09-18T12:02:10.960338+0000","last_clean":"2024-09-18T12:02:10.960338+0000","last_became_active":"2024-09-18T11:45:25.394579+0000","last_became_peered":"2024-09-18T11:45:25.394579+0000","last_unstale":"2024-09-18T12:02:10.960338+0000","last_undegraded":"2024-09-18T12:02:10.960338+0000","last_fullsized":"2024-09-18T12:02:10.960338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:59.330658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.914779+0000","last_change":"2024-09-18T11:45:25.405243+0000","last_active":"2024-09-18T12:02:10.914779+0000","last_peered":"2024-09-18T12:02:10.914779+0000","last_clean":"2024-09-18T12:02:10.914779+0000","last_became_active":"2024-09-18T11:45:25.403069+0000","last_became_peered":"2024-09-18T11:45:25.403069+0000","last_unstale":"2024-09-18T12:02:10.914779+0000","last_undegraded":"2024-09-18T12:02:10.914779+0000","last_fullsized":"2024-09-18T12:02:10.914779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:50:14.772996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607788,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315068,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056772,"statfs":{"total":96636764160,"available":96314134528,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.876,"15min":0.873},"min":{"1min":0.56799999999999995,"5min":0.48499999999999999,"15min":0.41499999999999998},"max":{"1min":1.1140000000000001,"5min":1.244,"15min":1.53},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.85199999999999998},"min":{"1min":0.70499999999999996,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0129999999999999,"5min":1.294,"15min":1.454},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85599999999999998,"15min":0.879},"min":{"1min":0.58299999999999996,"5min":0.52900000000000003,"15min":0.46500000000000002},"max":{"1min":0.999,"5min":1.2529999999999999,"15min":1.325},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.85999999999999999,"15min":0.86199999999999999},"min":{"1min":0.73799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.98899999999999999,"5min":1.208,"15min":1.2689999999999999},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79400000000000004,"15min":0.82199999999999995},"min":{"1min":0.70799999999999996,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":0.95699999999999996,"5min":1.2410000000000001,"15min":2.8900000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89000000000000001,"15min":0.86899999999999999},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.439},"max":{"1min":1.1060000000000001,"5min":1.6060000000000001,"15min":3.0760000000000001},"last":0.93200000000000005}]},{"osd":3,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.79100000000000004,"15min":0.77200000000000002},"min":{"1min":0.61199999999999999,"5min":0.45800000000000002,"15min":0.372},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.6950000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.56799999999999995,"5min":0.439,"15min":0.35299999999999998},"max":{"1min":1.198,"5min":1.204,"15min":1.446},"last":0.77200000000000002}]},{"osd":4,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.82799999999999996,"15min":0.86099999999999999},"min":{"1min":0.72899999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.159,"5min":1.159,"15min":1.323},"last":0.876},{"interface":"front","average":{"1min":0.995,"5min":0.98099999999999998,"15min":0.95999999999999996},"min":{"1min":0.82099999999999995,"5min":0.54600000000000004,"15min":0.439},"max":{"1min":1.232,"5min":1.3089999999999999,"15min":1.4670000000000001},"last":0.66800000000000004}]}]},{"osd":4,"up_from":12,"seq":51539607787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319244,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052596,"statfs":{"total":96636764160,"available":96309858304,"internally_reserved":0,"allocated":9613312,"data_stored":4953859,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.83699999999999997,"15min":0.82899999999999996},"min":{"1min":0.64800000000000002,"5min":0.51800000000000002,"15min":0.47699999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.3120000000000001},"last":1.04},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83399999999999996,"15min":0.83099999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.46400000000000002},"max":{"1min":1.131,"5min":1.272,"15min":1.296},"last":0.94199999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.504},"max":{"1min":0.92200000000000004,"5min":1.3169999999999999,"15min":1.331},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90500000000000003,"15min":0.85799999999999998},"min":{"1min":0.69199999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3600000000000001},"last":0.96099999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78800000000000003,"15min":0.82399999999999995},"min":{"1min":0.627,"5min":0.56799999999999995,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":1.0329999999999999,"15min":1.294},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.92800000000000005,"15min":0.90300000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.432},"max":{"1min":1.206,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.74299999999999999}]},{"osd":3,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.57999999999999996,"5min":0.48899999999999999,"15min":0.47499999999999998},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.879,"15min":0.89300000000000002},"min":{"1min":0.70599999999999996,"5min":0.5,"15min":0.44600000000000001},"max":{"1min":1.0900000000000001,"5min":1.4019999999999999,"15min":1.496},"last":0.82799999999999996}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.56999999999999995,"5min":0.53500000000000003,"15min":0.40100000000000002},"max":{"1min":1.085,"5min":1.4430000000000001,"15min":2.899},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0289999999999999,"15min":0.97199999999999998},"min":{"1min":0.76900000000000002,"5min":0.69299999999999995,"15min":0.56599999999999995},"max":{"1min":1.28,"5min":1.466,"15min":3.004},"last":0.995}]}]},{"osd":3,"up_from":12,"seq":51539607787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319292,"kb_used_data":9444,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052548,"statfs":{"total":96636764160,"available":96309809152,"internally_reserved":0,"allocated":9670656,"data_stored":5483791,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75,"15min":0.80500000000000005},"min":{"1min":0.55600000000000005,"5min":0.5,"15min":0.35999999999999999},"max":{"1min":1.2350000000000001,"5min":1.331,"15min":5.1390000000000002},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.82099999999999995,"15min":0.876},"min":{"1min":0.66700000000000004,"5min":0.56799999999999995,"15min":0.40999999999999998},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":7.0449999999999999},"last":0.58499999999999996}]},{"osd":1,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.79200000000000004,"5min":0.59699999999999998,"15min":0.505},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":7.4880000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93400000000000005,"15min":0.91500000000000004},"min":{"1min":0.69799999999999995,"5min":0.55000000000000004,"15min":0.45800000000000002},"max":{"1min":0.97999999999999998,"5min":1.325,"15min":4.9699999999999998},"last":0.88800000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.626,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":0.95499999999999996,"5min":1.419,"15min":5.2080000000000002},"last":0.626},{"interface":"front","average":{"1min":0.876,"5min":0.91900000000000004,"15min":0.96599999999999997},"min":{"1min":0.67600000000000005,"5min":0.63600000000000001,"15min":0.441},"max":{"1min":1.1259999999999999,"5min":1.3280000000000001,"15min":7.4180000000000001},"last":0.85099999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.73099999999999998},"min":{"1min":0.53400000000000003,"5min":0.49299999999999999,"15min":0.39600000000000002},"max":{"1min":0.94199999999999995,"5min":1.2569999999999999,"15min":4.7169999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97699999999999998},"min":{"1min":0.70499999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.361,"5min":1.7150000000000001,"15min":4.6260000000000003},"last":0.91800000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.82399999999999995,"15min":0.86499999999999999},"min":{"1min":0.621,"5min":0.54500000000000004,"15min":0.34699999999999998},"max":{"1min":1.159,"5min":1.208,"15min":7.4470000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0740000000000001,"15min":1.1160000000000001},"min":{"1min":0.871,"5min":0.68600000000000005,"15min":0.63200000000000001},"max":{"1min":1.4390000000000001,"5min":1.581,"15min":7.6269999999999998},"last":0.97499999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607787,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315084,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056756,"statfs":{"total":96636764160,"available":96314118144,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58499999999999996},"min":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.33100000000000002},"max":{"1min":0.98499999999999999,"5min":1.212,"15min":1.212},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72499999999999998},"min":{"1min":0.60999999999999999,"5min":0.48199999999999998,"15min":0.28699999999999998},"max":{"1min":1.02,"5min":1.254,"15min":1.254},"last":0.73899999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.495,"5min":0.41599999999999998,"15min":0.316},"max":{"1min":0.76100000000000001,"5min":0.93500000000000005,"15min":3.0899999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.56999999999999995,"5min":0.49299999999999999,"15min":0.41999999999999998},"max":{"1min":0.84199999999999997,"5min":1.577,"15min":4.1699999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.86599999999999999,"15min":0.89100000000000001},"min":{"1min":0.67800000000000005,"5min":0.52500000000000002,"15min":0.36899999999999999},"max":{"1min":1.0680000000000001,"5min":1.3799999999999999,"15min":2.6389999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.86199999999999999,"15min":0.88400000000000001},"min":{"1min":0.64300000000000002,"5min":0.623,"15min":0.53400000000000003},"max":{"1min":1.0029999999999999,"5min":1.2330000000000001,"15min":4.3310000000000004},"last":0.93100000000000005}]},{"osd":4,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.75900000000000001,"5min":0.67300000000000004,"15min":0.376},"max":{"1min":1.0600000000000001,"5min":1.2270000000000001,"15min":4.2270000000000003},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96999999999999997,"15min":0.94099999999999995},"min":{"1min":0.80800000000000005,"5min":0.69599999999999995,"15min":0.55700000000000005},"max":{"1min":1.0840000000000001,"5min":1.671,"15min":1.706},"last":0.90700000000000003}]},{"osd":5,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.89900000000000002,"15min":0.92000000000000004},"min":{"1min":0.71499999999999997,"5min":0.56999999999999995,"15min":0.39000000000000001},"max":{"1min":0.95099999999999996,"5min":2.4449999999999998,"15min":2.5960000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":0.96199999999999997},"min":{"1min":0.74299999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":0.97099999999999997,"5min":1.4890000000000001,"15min":4.7590000000000003},"last":0.84299999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315144,"kb_used_data":9440,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056696,"statfs":{"total":96636764160,"available":96314056704,"internally_reserved":0,"allocated":9666560,"data_stored":5483786,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55600000000000005,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.38100000000000001,"15min":0.29199999999999998},"max":{"1min":0.66700000000000004,"5min":0.82899999999999996,"15min":3.4900000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66800000000000004,"15min":0.68500000000000005},"min":{"1min":0.52800000000000002,"5min":0.442,"15min":0.27500000000000002},"max":{"1min":0.92700000000000005,"5min":1.0209999999999999,"15min":3.1019999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.67400000000000004},"min":{"1min":0.54900000000000004,"5min":0.443,"15min":0.317},"max":{"1min":0.94799999999999995,"5min":0.97299999999999998,"15min":3.5019999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81899999999999995,"15min":0.79000000000000004},"min":{"1min":0.66700000000000004,"5min":0.50700000000000001,"15min":0.41799999999999998},"max":{"1min":1.0620000000000001,"5min":1.0660000000000001,"15min":3.1779999999999999},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.69899999999999995,"5min":0.57899999999999996,"15min":0.40300000000000002},"max":{"1min":1.1120000000000001,"5min":1.2250000000000001,"15min":1.71},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88400000000000001,"15min":0.92100000000000004},"min":{"1min":0.67800000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.0229999999999999,"5min":1.26,"15min":4.3719999999999999},"last":0.70599999999999996}]},{"osd":4,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84599999999999997,"15min":0.876},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.48899999999999999},"max":{"1min":0.91500000000000004,"5min":1.123,"15min":2.4990000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.98899999999999999,"15min":0.93999999999999995},"min":{"1min":0.79500000000000004,"5min":0.70999999999999996,"15min":0.443},"max":{"1min":1.181,"5min":1.262,"15min":1.6679999999999999},"last":0.92300000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.90700000000000003,"15min":0.92400000000000004},"min":{"1min":0.79800000000000004,"5min":0.56100000000000005,"15min":0.504},"max":{"1min":1.0289999999999999,"5min":1.3300000000000001,"15min":2.5840000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.04,"5min":1.0209999999999999,"15min":0.97999999999999998},"min":{"1min":0.84799999999999998,"5min":0.67200000000000004,"15min":0.47399999999999998},"max":{"1min":1.2310000000000001,"5min":1.53,"15min":1.752},"last":1.02}]}]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315076,"kb_used_data":9396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056764,"statfs":{"total":96636764160,"available":96314126336,"internally_reserved":0,"allocated":9621504,"data_stored":4954882,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.191},"max":{"1min":0.69499999999999995,"5min":2.4470000000000001,"15min":3.8849999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.71799999999999997},"min":{"1min":0.54900000000000004,"5min":0.40300000000000002,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65700000000000003,"15min":0.67600000000000005},"min":{"1min":0.53200000000000003,"5min":0.501,"15min":0.23300000000000001},"max":{"1min":0.88100000000000001,"5min":1.3799999999999999,"15min":3.6030000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.871,"15min":0.81699999999999995},"min":{"1min":0.66600000000000004,"5min":0.63100000000000001,"15min":0.30399999999999999},"max":{"1min":1.0569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.89700000000000002}]},{"osd":3,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84899999999999998,"15min":0.84599999999999997},"min":{"1min":0.621,"5min":0.60099999999999998,"15min":0.28100000000000003},"max":{"1min":0.98099999999999998,"5min":2.2440000000000002,"15min":3.7909999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.93799999999999994,"15min":0.92500000000000004},"min":{"1min":0.69799999999999995,"5min":0.65800000000000003,"15min":0.313},"max":{"1min":1.248,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.0089999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.002,"15min":0.95299999999999996},"min":{"1min":0.82199999999999995,"5min":0.57199999999999995,"15min":0.40799999999999997},"max":{"1min":1.085,"5min":2.8530000000000002,"15min":3.6709999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96899999999999997,"15min":0.996},"min":{"1min":0.67900000000000005,"5min":0.67400000000000004,"15min":0.33700000000000002},"max":{"1min":1.175,"5min":2.726,"15min":3.5270000000000001},"last":1.111}]},{"osd":5,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88800000000000001},"min":{"1min":0.80800000000000005,"5min":0.66700000000000004,"15min":0.36799999999999999},"max":{"1min":1.0289999999999999,"5min":1.389,"15min":3.6469999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.98899999999999999,"15min":0.97499999999999998},"min":{"1min":0.71899999999999997,"5min":0.70099999999999996,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":2.7429999999999999,"15min":4.3280000000000003},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:38.648 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:38.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient: get_monmap_and_config 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient: build_initial_monmap 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient: monmap: 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:38.749936+0000 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:38.749936+0000 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:38.749 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 1 Processor -- start 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 1 -- start start 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient: init 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f75d80a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f75d80a0750 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f75d81385b0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d8097270 con 0x7f75d81385b0 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d80a3170 con 0x7f75d80a0750 2024-09-18T12:03:38.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d8096cb0 con 0x7f75d80a0380 2024-09-18T12:03:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d7fff700 10 monclient(hunting): authenticate will time out at 1990.591380s 2024-09-18T12:03:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f75d81385b0 0x7f75d813a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:51990/0 (socket says 172.21.15.112:51990) 2024-09-18T12:03:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d77fe700 1 -- 172.21.15.112:0/3573911510 learned_addr learned my addr 172.21.15.112:0/3573911510 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:38.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.745+0000 7f75d6ffd700 1 --1- 172.21.15.112:0/3573911510 >> v1:172.21.15.112:6790/0 conn(0x7f75d80a0380 0x7f75d8097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35684/0 (socket says 172.21.15.112:35684) 2024-09-18T12:03:38.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d77fe700 1 -- 172.21.15.112:0/3573911510 >> v1:172.21.15.112:6789/0 conn(0x7f75d81385b0 legacy=0x7f75d813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d67fc700 1 -- 172.21.15.112:0/3573911510 >> v1:172.21.15.129:6789/0 conn(0x7f75d80a0750 legacy=0x7f75d80a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d6ffd700 1 -- 172.21.15.112:0/3573911510 >> v1:172.21.15.112:6790/0 conn(0x7f75d80a0380 legacy=0x7f75d8097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736645804 0 0) 0x7f75d8097270 con 0x7f75d81385b0 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): my global_id is 164142 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c4003570 con 0x7f75d81385b0 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318386870 0 0) 0x7f75d8096cb0 con 0x7f75d80a0380 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): my global_id is 154241 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d8097270 con 0x7f75d80a0380 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110592409 0 0) 0x7f75d80a3170 con 0x7f75d80a0750 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient(hunting): my global_id is 144328 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d8096cb0 con 0x7f75d80a0750 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258367091 0 0) 0x7f75c4003570 con 0x7f75d81385b0 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 >> v1:172.21.15.112:6790/0 conn(0x7f75d80a0380 legacy=0x7f75d8097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 >> v1:172.21.15.129:6789/0 conn(0x7f75d80a0750 legacy=0x7f75d80a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 monclient: found mon.noname-a 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d80a1420 con 0x7f75d81385b0 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: _finish_auth 0 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d7fff700 5 monclient: authenticate success, global_id 164142 2024-09-18T12:03:38.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75d0001d80 con 0x7f75d81385b0 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: dump: 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.757 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75d0001f60 con 0x7f75d81385b0 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d7fff700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 1 -- 172.21.15.112:0/3573911510 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75d0004e10 con 0x7f75d81385b0 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d7fff700 4 set_mon_vals no callback set 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.749+0000 7f75d5ffb700 10 monclient: dump: 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.758 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.759 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient: shutdown 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- 172.21.15.112:0/3573911510 >> v1:172.21.15.112:6789/0 conn(0x7f75d81385b0 legacy=0x7f75d813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- 172.21.15.112:0/3573911510 shutdown_connections 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- 172.21.15.112:0/3573911510 >> 172.21.15.112:0/3573911510 conn(0x7f75d80a5990 msgr2=0x7f75d80a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- 172.21.15.112:0/3573911510 shutdown_connections 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- 172.21.15.112:0/3573911510 wait complete. 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient: build_initial_monmap 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient: monmap: 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:38.765 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 librados: starting msgr at 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 librados: starting objecter 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 Processor -- start 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- start start 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 librados: setting wanted keys 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 librados: calling monclient init 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient: init 2024-09-18T12:03:38.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f75d80a0380 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f75d80a0750 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f75d81385b0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d80a2010 con 0x7f75d80a0750 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d80980d0 con 0x7f75d80a0380 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d8120d30 con 0x7f75d81385b0 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d7fff700 10 monclient(hunting): authenticate will time out at 1990.598504s 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d67fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f75d80a0750 0x7f75d8146c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:52006/0 (socket says 172.21.15.112:52006) 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.753+0000 7f75d67fc700 1 -- 172.21.15.112:0/3599199531 learned_addr learned my addr 172.21.15.112:0/3599199531 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d77fe700 1 -- 172.21.15.112:0/3599199531 >> v1:172.21.15.112:6790/0 conn(0x7f75d81385b0 legacy=0x7f75d814a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 client.?.objecter ms_handle_connect 0x7f75d81385b0 2024-09-18T12:03:38.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d67fc700 1 -- 172.21.15.112:0/3599199531 >> v1:172.21.15.112:6789/0 conn(0x7f75d80a0750 legacy=0x7f75d8146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 client.?.objecter ms_handle_connect 0x7f75d80a0750 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 client.?.objecter ms_handle_connect 0x7f75d80a0380 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d6ffd700 1 -- 172.21.15.112:0/3599199531 >> v1:172.21.15.129:6789/0 conn(0x7f75d80a0380 legacy=0x7f75d8141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759275495 0 0) 0x7f75d8120d30 con 0x7f75d81385b0 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): my global_id is 154244 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8004f10 con 0x7f75d81385b0 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608819416 0 0) 0x7f75d80a2010 con 0x7f75d80a0750 2024-09-18T12:03:38.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): my global_id is 164145 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d8120d30 con 0x7f75d80a0750 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195231979 0 0) 0x7f75d80980d0 con 0x7f75d80a0380 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): my global_id is 144331 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d80a2010 con 0x7f75d80a0380 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1773909047 0 0) 0x7f75b8004f10 con 0x7f75d81385b0 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75d80980d0 con 0x7f75d81385b0 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75d0004d50 con 0x7f75d81385b0 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): dump: 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 255086327 0 0) 0x7f75d8120d30 con 0x7f75d80a0750 2024-09-18T12:03:38.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75b8004f10 con 0x7f75d80a0750 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75c0001cf0 con 0x7f75d80a0750 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): dump: 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2121035673 0 0) 0x7f75d80a2010 con 0x7f75d80a0380 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75d8120d30 con 0x7f75d80a0380 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75c8001b20 con 0x7f75d80a0380 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient(hunting): dump: 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.772 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1114128083 0 0) 0x7f75d80980d0 con 0x7f75d81385b0 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 >> v1:172.21.15.129:6789/0 conn(0x7f75d80a0380 legacy=0x7f75d8141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 >> v1:172.21.15.112:6789/0 conn(0x7f75d80a0750 legacy=0x7f75d8146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient: _finish_hunting 0 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 monclient: found mon.c 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8123470 con 0x7f75d81385b0 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient: _finish_auth 0 2024-09-18T12:03:38.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 5 monclient: authenticate success, global_id 154244 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 10 monclient: _renew_subs 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d8120820 con 0x7f75d81385b0 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 10 client.154244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 10 monclient: _renew_subs 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d81213f0 con 0x7f75d81385b0 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d7fff700 1 librados: init done 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75d00016e0 con 0x7f75d81385b0 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f75d00054e0 con 0x7f75d81385b0 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75d57fa700 4 set_mon_vals no callback set 2024-09-18T12:03:38.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.757+0000 7f75ceffd700 10 monclient: dump: 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f75d000fd60 con 0x7f75d81385b0 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75d7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75d7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:38.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75bc0050b0 con 0x7f75d81385b0 2024-09-18T12:03:38.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f75d0043ee0 con 0x7f75d81385b0 2024-09-18T12:03:38.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75ceffd700 10 client.154244.objecter ms_dispatch 0x7f75d8000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:38.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75ceffd700 3 client.154244.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:38.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75ceffd700 3 client.154244.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:38.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75ceffd700 20 client.154244.objecter dump_active .. 0 homeless 2024-09-18T12:03:38.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75d6ffd700 10 monclient: get_auth_request con 0x7f75b8038580 auth_method 0 2024-09-18T12:03:38.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.761+0000 7f75ceffd700 10 client.154244.objecter ms_handle_connect 0x7f75b8038580 2024-09-18T12:03:38.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.769+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f75d0014e60 con 0x7f75d81385b0 2024-09-18T12:03:38.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.769+0000 7f75ceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:38.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.769+0000 7f75ceffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:38.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.929+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f75bc000b90 con 0x7f75b8038580 2024-09-18T12:03:38.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75ceffd700 1 -- 172.21.15.112:0/3599199531 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134033 (unknown 2965378022 0 767431492) 0x7f75bc000b90 con 0x7f75b8038580 2024-09-18T12:03:38.944 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T12:03:38.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 10 librados: watch_flush enter 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 10 librados: watch_flush exit 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 20 client.154244.objecter shutdown clearing up homeless session... 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 10 client.154244.objecter successfully canceled tick 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 >> v1:172.21.15.129:6812/20806 conn(0x7f75b8038580 legacy=0x7f75b803aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 10 monclient: shutdown 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 >> v1:172.21.15.112:6790/0 conn(0x7f75d81385b0 legacy=0x7f75d814a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 shutdown_connections 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 >> 172.21.15.112:0/3599199531 conn(0x7f75d80a5990 msgr2=0x7f75d80a4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 shutdown_connections 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 1 -- 172.21.15.112:0/3599199531 wait complete. 2024-09-18T12:03:38.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:38.941+0000 7f75d7fff700 1 librados: shutdown 2024-09-18T12:03:38.947 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T12:03:38.958 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1274,"stamp":"2024-09-18T12:03:38.380870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1898908,"kb_used_data":56444,"kb_used_omap":9,"kb_used_meta":1842294,"kb_avail":564332132,"statfs":{"total":579820584960,"available":577876103168,"internally_reserved":0,"allocated":57798656,"data_stored":30786034,"data_compressed":320902,"data_compressed_allocated":876544,"data_compressed_original":2744320,"omap_allocated":10224,"internal_metadata":1886509072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003849"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522453+0000","last_change":"2024-09-18T11:45:25.374991+0000","last_active":"2024-09-18T12:02:11.522453+0000","last_peered":"2024-09-18T12:02:11.522453+0000","last_clean":"2024-09-18T12:02:11.522453+0000","last_became_active":"2024-09-18T11:45:25.373642+0000","last_became_peered":"2024-09-18T11:45:25.373642+0000","last_unstale":"2024-09-18T12:02:11.522453+0000","last_undegraded":"2024-09-18T12:02:11.522453+0000","last_fullsized":"2024-09-18T12:02:11.522453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:57:36.040393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519515+0000","last_change":"2024-09-18T11:45:25.329845+0000","last_active":"2024-09-18T12:02:11.519515+0000","last_peered":"2024-09-18T12:02:11.519515+0000","last_clean":"2024-09-18T12:02:11.519515+0000","last_became_active":"2024-09-18T11:45:25.328032+0000","last_became_peered":"2024-09-18T11:45:25.328032+0000","last_unstale":"2024-09-18T12:02:11.519515+0000","last_undegraded":"2024-09-18T12:02:11.519515+0000","last_fullsized":"2024-09-18T12:02:11.519515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:13.680321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913257+0000","last_change":"2024-09-18T11:45:25.306434+0000","last_active":"2024-09-18T12:02:10.913257+0000","last_peered":"2024-09-18T12:02:10.913257+0000","last_clean":"2024-09-18T12:02:10.913257+0000","last_became_active":"2024-09-18T11:45:25.305872+0000","last_became_peered":"2024-09-18T11:45:25.305872+0000","last_unstale":"2024-09-18T12:02:10.913257+0000","last_undegraded":"2024-09-18T12:02:10.913257+0000","last_fullsized":"2024-09-18T12:02:10.913257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:30:00.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519584+0000","last_change":"2024-09-18T11:45:25.411469+0000","last_active":"2024-09-18T12:02:11.519584+0000","last_peered":"2024-09-18T12:02:11.519584+0000","last_clean":"2024-09-18T12:02:11.519584+0000","last_became_active":"2024-09-18T11:45:25.411133+0000","last_became_peered":"2024-09-18T11:45:25.411133+0000","last_unstale":"2024-09-18T12:02:11.519584+0000","last_undegraded":"2024-09-18T12:02:11.519584+0000","last_fullsized":"2024-09-18T12:02:11.519584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:02:10.494289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523123+0000","last_change":"2024-09-18T11:45:25.400321+0000","last_active":"2024-09-18T12:02:11.523123+0000","last_peered":"2024-09-18T12:02:11.523123+0000","last_clean":"2024-09-18T12:02:11.523123+0000","last_became_active":"2024-09-18T11:45:25.399898+0000","last_became_peered":"2024-09-18T11:45:25.399898+0000","last_unstale":"2024-09-18T12:02:11.523123+0000","last_undegraded":"2024-09-18T12:02:11.523123+0000","last_fullsized":"2024-09-18T12:02:11.523123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:14.033638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.520050+0000","last_change":"2024-09-18T11:45:25.397202+0000","last_active":"2024-09-18T12:02:11.520050+0000","last_peered":"2024-09-18T12:02:11.520050+0000","last_clean":"2024-09-18T12:02:11.520050+0000","last_became_active":"2024-09-18T11:45:25.396975+0000","last_became_peered":"2024-09-18T11:45:25.396975+0000","last_unstale":"2024-09-18T12:02:11.520050+0000","last_undegraded":"2024-09-18T12:02:11.520050+0000","last_fullsized":"2024-09-18T12:02:11.520050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:37:37.634734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.910998+0000","last_change":"2024-09-18T11:45:25.341093+0000","last_active":"2024-09-18T12:02:10.910998+0000","last_peered":"2024-09-18T12:02:10.910998+0000","last_clean":"2024-09-18T12:02:10.910998+0000","last_became_active":"2024-09-18T11:45:25.340726+0000","last_became_peered":"2024-09-18T11:45:25.340726+0000","last_unstale":"2024-09-18T12:02:10.910998+0000","last_undegraded":"2024-09-18T12:02:10.910998+0000","last_fullsized":"2024-09-18T12:02:10.910998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:36:59.172751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960537+0000","last_change":"2024-09-18T11:45:25.408737+0000","last_active":"2024-09-18T12:02:10.960537+0000","last_peered":"2024-09-18T12:02:10.960537+0000","last_clean":"2024-09-18T12:02:10.960537+0000","last_became_active":"2024-09-18T11:45:25.408464+0000","last_became_peered":"2024-09-18T11:45:25.408464+0000","last_unstale":"2024-09-18T12:02:10.960537+0000","last_undegraded":"2024-09-18T12:02:10.960537+0000","last_fullsized":"2024-09-18T12:02:10.960537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:44:57.076431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.909867+0000","last_change":"2024-09-18T11:45:25.344531+0000","last_active":"2024-09-18T12:02:10.909867+0000","last_peered":"2024-09-18T12:02:10.909867+0000","last_clean":"2024-09-18T12:02:10.909867+0000","last_became_active":"2024-09-18T11:45:25.343097+0000","last_became_peered":"2024-09-18T11:45:25.343097+0000","last_unstale":"2024-09-18T12:02:10.909867+0000","last_undegraded":"2024-09-18T12:02:10.909867+0000","last_fullsized":"2024-09-18T12:02:10.909867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:49.438992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519649+0000","last_change":"2024-09-18T11:45:25.377493+0000","last_active":"2024-09-18T12:02:11.519649+0000","last_peered":"2024-09-18T12:02:11.519649+0000","last_clean":"2024-09-18T12:02:11.519649+0000","last_became_active":"2024-09-18T11:45:25.377194+0000","last_became_peered":"2024-09-18T11:45:25.377194+0000","last_unstale":"2024-09-18T12:02:11.519649+0000","last_undegraded":"2024-09-18T12:02:11.519649+0000","last_fullsized":"2024-09-18T12:02:11.519649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:44:53.107224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522770+0000","last_change":"2024-09-18T11:45:25.346404+0000","last_active":"2024-09-18T12:02:11.522770+0000","last_peered":"2024-09-18T12:02:11.522770+0000","last_clean":"2024-09-18T12:02:11.522770+0000","last_became_active":"2024-09-18T11:45:25.341682+0000","last_became_peered":"2024-09-18T11:45:25.341682+0000","last_unstale":"2024-09-18T12:02:11.522770+0000","last_undegraded":"2024-09-18T12:02:11.522770+0000","last_fullsized":"2024-09-18T12:02:11.522770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:04.010096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522588+0000","last_change":"2024-09-18T11:45:25.383080+0000","last_active":"2024-09-18T12:02:11.522588+0000","last_peered":"2024-09-18T12:02:11.522588+0000","last_clean":"2024-09-18T12:02:11.522588+0000","last_became_active":"2024-09-18T11:45:25.377663+0000","last_became_peered":"2024-09-18T11:45:25.377663+0000","last_unstale":"2024-09-18T12:02:11.522588+0000","last_undegraded":"2024-09-18T12:02:11.522588+0000","last_fullsized":"2024-09-18T12:02:11.522588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:46:12.443901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":649,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.914509+0000","last_change":"2024-09-18T11:44:24.299141+0000","last_active":"2024-09-18T12:02:10.914509+0000","last_peered":"2024-09-18T12:02:10.914509+0000","last_clean":"2024-09-18T12:02:10.914509+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T12:02:10.914509+0000","last_undegraded":"2024-09-18T12:02:10.914509+0000","last_fullsized":"2024-09-18T12:02:10.914509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:35:52.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960533+0000","last_change":"2024-09-18T11:45:25.405730+0000","last_active":"2024-09-18T12:02:10.960533+0000","last_peered":"2024-09-18T12:02:10.960533+0000","last_clean":"2024-09-18T12:02:10.960533+0000","last_became_active":"2024-09-18T11:45:25.405515+0000","last_became_peered":"2024-09-18T11:45:25.405515+0000","last_unstale":"2024-09-18T12:02:10.960533+0000","last_undegraded":"2024-09-18T12:02:10.960533+0000","last_fullsized":"2024-09-18T12:02:10.960533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:39:09.362580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911684+0000","last_change":"2024-09-18T11:44:24.208317+0000","last_active":"2024-09-18T12:02:10.911684+0000","last_peered":"2024-09-18T12:02:10.911684+0000","last_clean":"2024-09-18T12:02:10.911684+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T12:02:10.911684+0000","last_undegraded":"2024-09-18T12:02:10.911684+0000","last_fullsized":"2024-09-18T12:02:10.911684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:54.979951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913034+0000","last_change":"2024-09-18T11:45:25.306008+0000","last_active":"2024-09-18T12:02:10.913034+0000","last_peered":"2024-09-18T12:02:10.913034+0000","last_clean":"2024-09-18T12:02:10.913034+0000","last_became_active":"2024-09-18T11:45:25.305686+0000","last_became_peered":"2024-09-18T11:45:25.305686+0000","last_unstale":"2024-09-18T12:02:10.913034+0000","last_undegraded":"2024-09-18T12:02:10.913034+0000","last_fullsized":"2024-09-18T12:02:10.913034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:31:41.349569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519991+0000","last_change":"2024-09-18T11:44:24.213224+0000","last_active":"2024-09-18T12:02:11.519991+0000","last_peered":"2024-09-18T12:02:11.519991+0000","last_clean":"2024-09-18T12:02:11.519991+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T12:02:11.519991+0000","last_undegraded":"2024-09-18T12:02:11.519991+0000","last_fullsized":"2024-09-18T12:02:11.519991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:47.310939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"20'7","reported_seq":660,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.910044+0000","last_change":"2024-09-18T11:45:25.403798+0000","last_active":"2024-09-18T12:02:10.910044+0000","last_peered":"2024-09-18T12:02:10.910044+0000","last_clean":"2024-09-18T12:02:10.910044+0000","last_became_active":"2024-09-18T11:45:25.403586+0000","last_became_peered":"2024-09-18T11:45:25.403586+0000","last_unstale":"2024-09-18T12:02:10.910044+0000","last_undegraded":"2024-09-18T12:02:10.910044+0000","last_fullsized":"2024-09-18T12:02:10.910044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:53:41.121574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":654,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522879+0000","last_change":"2024-09-18T11:44:24.507933+0000","last_active":"2024-09-18T12:02:11.522879+0000","last_peered":"2024-09-18T12:02:11.522879+0000","last_clean":"2024-09-18T12:02:11.522879+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T12:02:11.522879+0000","last_undegraded":"2024-09-18T12:02:11.522879+0000","last_fullsized":"2024-09-18T12:02:11.522879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:56.697013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522889+0000","last_change":"2024-09-18T11:45:25.369556+0000","last_active":"2024-09-18T12:02:11.522889+0000","last_peered":"2024-09-18T12:02:11.522889+0000","last_clean":"2024-09-18T12:02:11.522889+0000","last_became_active":"2024-09-18T11:45:25.367362+0000","last_became_peered":"2024-09-18T11:45:25.367362+0000","last_unstale":"2024-09-18T12:02:11.522889+0000","last_undegraded":"2024-09-18T12:02:11.522889+0000","last_fullsized":"2024-09-18T12:02:11.522889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:23:17.811113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":743,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.907687+0000","last_change":"2024-09-18T11:44:21.097910+0000","last_active":"2024-09-18T12:02:10.907687+0000","last_peered":"2024-09-18T12:02:10.907687+0000","last_clean":"2024-09-18T12:02:10.907687+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T12:02:10.907687+0000","last_undegraded":"2024-09-18T12:02:10.907687+0000","last_fullsized":"2024-09-18T12:02:10.907687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:48.394073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":655,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522634+0000","last_change":"2024-09-18T11:44:24.508079+0000","last_active":"2024-09-18T12:02:11.522634+0000","last_peered":"2024-09-18T12:02:11.522634+0000","last_clean":"2024-09-18T12:02:11.522634+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T12:02:11.522634+0000","last_undegraded":"2024-09-18T12:02:11.522634+0000","last_fullsized":"2024-09-18T12:02:11.522634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:41:30.648862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913627+0000","last_change":"2024-09-18T11:45:25.407247+0000","last_active":"2024-09-18T12:02:10.913627+0000","last_peered":"2024-09-18T12:02:10.913627+0000","last_clean":"2024-09-18T12:02:10.913627+0000","last_became_active":"2024-09-18T11:45:25.406430+0000","last_became_peered":"2024-09-18T11:45:25.406430+0000","last_unstale":"2024-09-18T12:02:10.913627+0000","last_undegraded":"2024-09-18T12:02:10.913627+0000","last_fullsized":"2024-09-18T12:02:10.913627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:56:03.017626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"17'2","reported_seq":660,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519718+0000","last_change":"2024-09-18T11:44:24.214482+0000","last_active":"2024-09-18T12:02:11.519718+0000","last_peered":"2024-09-18T12:02:11.519718+0000","last_clean":"2024-09-18T12:02:11.519718+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T12:02:11.519718+0000","last_undegraded":"2024-09-18T12:02:11.519718+0000","last_fullsized":"2024-09-18T12:02:11.519718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:26:34.055867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523238+0000","last_change":"2024-09-18T11:45:25.365786+0000","last_active":"2024-09-18T12:02:11.523238+0000","last_peered":"2024-09-18T12:02:11.523238+0000","last_clean":"2024-09-18T12:02:11.523238+0000","last_became_active":"2024-09-18T11:45:25.363235+0000","last_became_peered":"2024-09-18T11:45:25.363235+0000","last_unstale":"2024-09-18T12:02:11.523238+0000","last_undegraded":"2024-09-18T12:02:11.523238+0000","last_fullsized":"2024-09-18T12:02:11.523238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:15:20.066742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":649,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.912589+0000","last_change":"2024-09-18T11:44:24.298975+0000","last_active":"2024-09-18T12:02:10.912589+0000","last_peered":"2024-09-18T12:02:10.912589+0000","last_clean":"2024-09-18T12:02:10.912589+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T12:02:10.912589+0000","last_undegraded":"2024-09-18T12:02:10.912589+0000","last_fullsized":"2024-09-18T12:02:10.912589+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:27.175495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960440+0000","last_change":"2024-09-18T11:45:25.356461+0000","last_active":"2024-09-18T12:02:10.960440+0000","last_peered":"2024-09-18T12:02:10.960440+0000","last_clean":"2024-09-18T12:02:10.960440+0000","last_became_active":"2024-09-18T11:45:25.356178+0000","last_became_peered":"2024-09-18T11:45:25.356178+0000","last_unstale":"2024-09-18T12:02:10.960440+0000","last_undegraded":"2024-09-18T12:02:10.960440+0000","last_fullsized":"2024-09-18T12:02:10.960440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:06:26.698280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.908011+0000","last_change":"2024-09-18T11:44:24.208252+0000","last_active":"2024-09-18T12:02:10.908011+0000","last_peered":"2024-09-18T12:02:10.908011+0000","last_clean":"2024-09-18T12:02:10.908011+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T12:02:10.908011+0000","last_undegraded":"2024-09-18T12:02:10.908011+0000","last_fullsized":"2024-09-18T12:02:10.908011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:08:27.257954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.520056+0000","last_change":"2024-09-18T11:45:25.396966+0000","last_active":"2024-09-18T12:02:11.520056+0000","last_peered":"2024-09-18T12:02:11.520056+0000","last_clean":"2024-09-18T12:02:11.520056+0000","last_became_active":"2024-09-18T11:45:25.396428+0000","last_became_peered":"2024-09-18T11:45:25.396428+0000","last_unstale":"2024-09-18T12:02:11.520056+0000","last_undegraded":"2024-09-18T12:02:11.520056+0000","last_fullsized":"2024-09-18T12:02:11.520056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:08:22.504624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911592+0000","last_change":"2024-09-18T11:45:25.399933+0000","last_active":"2024-09-18T12:02:10.911592+0000","last_peered":"2024-09-18T12:02:10.911592+0000","last_clean":"2024-09-18T12:02:10.911592+0000","last_became_active":"2024-09-18T11:45:25.399695+0000","last_became_peered":"2024-09-18T11:45:25.399695+0000","last_unstale":"2024-09-18T12:02:10.911592+0000","last_undegraded":"2024-09-18T12:02:10.911592+0000","last_fullsized":"2024-09-18T12:02:10.911592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:01:15.698698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913437+0000","last_change":"2024-09-18T11:45:25.387334+0000","last_active":"2024-09-18T12:02:10.913437+0000","last_peered":"2024-09-18T12:02:10.913437+0000","last_clean":"2024-09-18T12:02:10.913437+0000","last_became_active":"2024-09-18T11:45:25.386932+0000","last_became_peered":"2024-09-18T11:45:25.386932+0000","last_unstale":"2024-09-18T12:02:10.913437+0000","last_undegraded":"2024-09-18T12:02:10.913437+0000","last_fullsized":"2024-09-18T12:02:10.913437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:34:23.587779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522768+0000","last_change":"2024-09-18T11:45:25.403356+0000","last_active":"2024-09-18T12:02:11.522768+0000","last_peered":"2024-09-18T12:02:11.522768+0000","last_clean":"2024-09-18T12:02:11.522768+0000","last_became_active":"2024-09-18T11:45:25.401979+0000","last_became_peered":"2024-09-18T11:45:25.401979+0000","last_unstale":"2024-09-18T12:02:11.522768+0000","last_undegraded":"2024-09-18T12:02:11.522768+0000","last_fullsized":"2024-09-18T12:02:11.522768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:29:14.433201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523103+0000","last_change":"2024-09-18T11:45:25.381976+0000","last_active":"2024-09-18T12:02:11.523103+0000","last_peered":"2024-09-18T12:02:11.523103+0000","last_clean":"2024-09-18T12:02:11.523103+0000","last_became_active":"2024-09-18T11:45:25.381652+0000","last_became_peered":"2024-09-18T11:45:25.381652+0000","last_unstale":"2024-09-18T12:02:11.523103+0000","last_undegraded":"2024-09-18T12:02:11.523103+0000","last_fullsized":"2024-09-18T12:02:11.523103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:21.420291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522937+0000","last_change":"2024-09-18T11:45:25.381748+0000","last_active":"2024-09-18T12:02:11.522937+0000","last_peered":"2024-09-18T12:02:11.522937+0000","last_clean":"2024-09-18T12:02:11.522937+0000","last_became_active":"2024-09-18T11:45:25.379657+0000","last_became_peered":"2024-09-18T11:45:25.379657+0000","last_unstale":"2024-09-18T12:02:11.522937+0000","last_undegraded":"2024-09-18T12:02:11.522937+0000","last_fullsized":"2024-09-18T12:02:11.522937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:44:38.923827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523379+0000","last_change":"2024-09-18T11:45:25.340204+0000","last_active":"2024-09-18T12:02:11.523379+0000","last_peered":"2024-09-18T12:02:11.523379+0000","last_clean":"2024-09-18T12:02:11.523379+0000","last_became_active":"2024-09-18T11:45:25.338772+0000","last_became_peered":"2024-09-18T11:45:25.338772+0000","last_unstale":"2024-09-18T12:02:11.523379+0000","last_undegraded":"2024-09-18T12:02:11.523379+0000","last_fullsized":"2024-09-18T12:02:11.523379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:09.409329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522520+0000","last_change":"2024-09-18T11:45:25.369783+0000","last_active":"2024-09-18T12:02:11.522520+0000","last_peered":"2024-09-18T12:02:11.522520+0000","last_clean":"2024-09-18T12:02:11.522520+0000","last_became_active":"2024-09-18T11:45:25.369294+0000","last_became_peered":"2024-09-18T11:45:25.369294+0000","last_unstale":"2024-09-18T12:02:11.522520+0000","last_undegraded":"2024-09-18T12:02:11.522520+0000","last_fullsized":"2024-09-18T12:02:11.522520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:20:30.771482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.912743+0000","last_change":"2024-09-18T11:45:25.393624+0000","last_active":"2024-09-18T12:02:10.912743+0000","last_peered":"2024-09-18T12:02:10.912743+0000","last_clean":"2024-09-18T12:02:10.912743+0000","last_became_active":"2024-09-18T11:45:25.393093+0000","last_became_peered":"2024-09-18T11:45:25.393093+0000","last_unstale":"2024-09-18T12:02:10.912743+0000","last_undegraded":"2024-09-18T12:02:10.912743+0000","last_fullsized":"2024-09-18T12:02:10.912743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:17:30.506236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911484+0000","last_change":"2024-09-18T11:45:25.382303+0000","last_active":"2024-09-18T12:02:10.911484+0000","last_peered":"2024-09-18T12:02:10.911484+0000","last_clean":"2024-09-18T12:02:10.911484+0000","last_became_active":"2024-09-18T11:45:25.380362+0000","last_became_peered":"2024-09-18T11:45:25.380362+0000","last_unstale":"2024-09-18T12:02:10.911484+0000","last_undegraded":"2024-09-18T12:02:10.911484+0000","last_fullsized":"2024-09-18T12:02:10.911484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:11:31.985540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523224+0000","last_change":"2024-09-18T11:45:25.358090+0000","last_active":"2024-09-18T12:02:11.523224+0000","last_peered":"2024-09-18T12:02:11.523224+0000","last_clean":"2024-09-18T12:02:11.523224+0000","last_became_active":"2024-09-18T11:45:25.357477+0000","last_became_peered":"2024-09-18T11:45:25.357477+0000","last_unstale":"2024-09-18T12:02:11.523224+0000","last_undegraded":"2024-09-18T12:02:11.523224+0000","last_fullsized":"2024-09-18T12:02:11.523224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:54:53.792263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960338+0000","last_change":"2024-09-18T11:45:25.394821+0000","last_active":"2024-09-18T12:02:10.960338+0000","last_peered":"2024-09-18T12:02:10.960338+0000","last_clean":"2024-09-18T12:02:10.960338+0000","last_became_active":"2024-09-18T11:45:25.394579+0000","last_became_peered":"2024-09-18T11:45:25.394579+0000","last_unstale":"2024-09-18T12:02:10.960338+0000","last_undegraded":"2024-09-18T12:02:10.960338+0000","last_fullsized":"2024-09-18T12:02:10.960338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:59.330658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.914779+0000","last_change":"2024-09-18T11:45:25.405243+0000","last_active":"2024-09-18T12:02:10.914779+0000","last_peered":"2024-09-18T12:02:10.914779+0000","last_clean":"2024-09-18T12:02:10.914779+0000","last_became_active":"2024-09-18T11:45:25.403069+0000","last_became_peered":"2024-09-18T11:45:25.403069+0000","last_unstale":"2024-09-18T12:02:10.914779+0000","last_undegraded":"2024-09-18T12:02:10.914779+0000","last_fullsized":"2024-09-18T12:02:10.914779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:50:14.772996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607788,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315068,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056772,"statfs":{"total":96636764160,"available":96314134528,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.876,"15min":0.873},"min":{"1min":0.56799999999999995,"5min":0.48499999999999999,"15min":0.41499999999999998},"max":{"1min":1.1140000000000001,"5min":1.244,"15min":1.53},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.85199999999999998},"min":{"1min":0.70499999999999996,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0129999999999999,"5min":1.294,"15min":1.454},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85599999999999998,"15min":0.879},"min":{"1min":0.58299999999999996,"5min":0.52900000000000003,"15min":0.46500000000000002},"max":{"1min":0.999,"5min":1.2529999999999999,"15min":1.325},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.85999999999999999,"15min":0.86199999999999999},"min":{"1min":0.73799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.98899999999999999,"5min":1.208,"15min":1.2689999999999999},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79400000000000004,"15min":0.82199999999999995},"min":{"1min":0.70799999999999996,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":0.95699999999999996,"5min":1.2410000000000001,"15min":2.8900000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89000000000000001,"15min":0.86899999999999999},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.439},"max":{"1min":1.1060000000000001,"5min":1.6060000000000001,"15min":3.0760000000000001},"last":0.93200000000000005}]},{"osd":3,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.79100000000000004,"15min":0.77200000000000002},"min":{"1min":0.61199999999999999,"5min":0.45800000000000002,"15min":0.372},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.6950000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.56799999999999995,"5min":0.439,"15min":0.35299999999999998},"max":{"1min":1.198,"5min":1.204,"15min":1.446},"last":0.77200000000000002}]},{"osd":4,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.82799999999999996,"15min":0.86099999999999999},"min":{"1min":0.72899999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.159,"5min":1.159,"15min":1.323},"last":0.876},{"interface":"front","average":{"1min":0.995,"5min":0.98099999999999998,"15min":0.95999999999999996},"min":{"1min":0.82099999999999995,"5min":0.54600000000000004,"15min":0.439},"max":{"1min":1.232,"5min":1.3089999999999999,"15min":1.4670000000000001},"last":0.66800000000000004}]}]},{"osd":4,"up_from":12,"seq":51539607787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319244,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052596,"statfs":{"total":96636764160,"available":96309858304,"internally_reserved":0,"allocated":9613312,"data_stored":4953859,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.83699999999999997,"15min":0.82899999999999996},"min":{"1min":0.64800000000000002,"5min":0.51800000000000002,"15min":0.47699999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.3120000000000001},"last":1.04},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83399999999999996,"15min":0.83099999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.46400000000000002},"max":{"1min":1.131,"5min":1.272,"15min":1.296},"last":0.94199999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.504},"max":{"1min":0.92200000000000004,"5min":1.3169999999999999,"15min":1.331},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90500000000000003,"15min":0.85799999999999998},"min":{"1min":0.69199999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3600000000000001},"last":0.96099999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78800000000000003,"15min":0.82399999999999995},"min":{"1min":0.627,"5min":0.56799999999999995,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":1.0329999999999999,"15min":1.294},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.92800000000000005,"15min":0.90300000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.432},"max":{"1min":1.206,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.74299999999999999}]},{"osd":3,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.57999999999999996,"5min":0.48899999999999999,"15min":0.47499999999999998},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.879,"15min":0.89300000000000002},"min":{"1min":0.70599999999999996,"5min":0.5,"15min":0.44600000000000001},"max":{"1min":1.0900000000000001,"5min":1.4019999999999999,"15min":1.496},"last":0.82799999999999996}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.56999999999999995,"5min":0.53500000000000003,"15min":0.40100000000000002},"max":{"1min":1.085,"5min":1.4430000000000001,"15min":2.899},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0289999999999999,"15min":0.97199999999999998},"min":{"1min":0.76900000000000002,"5min":0.69299999999999995,"15min":0.56599999999999995},"max":{"1min":1.28,"5min":1.466,"15min":3.004},"last":0.995}]}]},{"osd":3,"up_from":12,"seq":51539607787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319292,"kb_used_data":9444,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052548,"statfs":{"total":96636764160,"available":96309809152,"internally_reserved":0,"allocated":9670656,"data_stored":5483791,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75,"15min":0.80500000000000005},"min":{"1min":0.55600000000000005,"5min":0.5,"15min":0.35999999999999999},"max":{"1min":1.2350000000000001,"5min":1.331,"15min":5.1390000000000002},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.82099999999999995,"15min":0.876},"min":{"1min":0.66700000000000004,"5min":0.56799999999999995,"15min":0.40999999999999998},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":7.0449999999999999},"last":0.58499999999999996}]},{"osd":1,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.79200000000000004,"5min":0.59699999999999998,"15min":0.505},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":7.4880000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93400000000000005,"15min":0.91500000000000004},"min":{"1min":0.69799999999999995,"5min":0.55000000000000004,"15min":0.45800000000000002},"max":{"1min":0.97999999999999998,"5min":1.325,"15min":4.9699999999999998},"last":0.88800000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.626,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":0.95499999999999996,"5min":1.419,"15min":5.2080000000000002},"last":0.626},{"interface":"front","average":{"1min":0.876,"5min":0.91900000000000004,"15min":0.96599999999999997},"min":{"1min":0.67600000000000005,"5min":0.63600000000000001,"15min":0.441},"max":{"1min":1.1259999999999999,"5min":1.3280000000000001,"15min":7.4180000000000001},"last":0.85099999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.73099999999999998},"min":{"1min":0.53400000000000003,"5min":0.49299999999999999,"15min":0.39600000000000002},"max":{"1min":0.94199999999999995,"5min":1.2569999999999999,"15min":4.7169999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97699999999999998},"min":{"1min":0.70499999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.361,"5min":1.7150000000000001,"15min":4.6260000000000003},"last":0.91800000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.82399999999999995,"15min":0.86499999999999999},"min":{"1min":0.621,"5min":0.54500000000000004,"15min":0.34699999999999998},"max":{"1min":1.159,"5min":1.208,"15min":7.4470000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0740000000000001,"15min":1.1160000000000001},"min":{"1min":0.871,"5min":0.68600000000000005,"15min":0.63200000000000001},"max":{"1min":1.4390000000000001,"5min":1.581,"15min":7.6269999999999998},"last":0.97499999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607787,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315084,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056756,"statfs":{"total":96636764160,"available":96314118144,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58499999999999996},"min":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.33100000000000002},"max":{"1min":0.98499999999999999,"5min":1.212,"15min":1.212},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72499999999999998},"min":{"1min":0.60999999999999999,"5min":0.48199999999999998,"15min":0.28699999999999998},"max":{"1min":1.02,"5min":1.254,"15min":1.254},"last":0.73899999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.495,"5min":0.41599999999999998,"15min":0.316},"max":{"1min":0.76100000000000001,"5min":0.93500000000000005,"15min":3.0899999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.56999999999999995,"5min":0.49299999999999999,"15min":0.41999999999999998},"max":{"1min":0.84199999999999997,"5min":1.577,"15min":4.1699999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.86599999999999999,"15min":0.89100000000000001},"min":{"1min":0.67800000000000005,"5min":0.52500000000000002,"15min":0.36899999999999999},"max":{"1min":1.0680000000000001,"5min":1.3799999999999999,"15min":2.6389999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.86199999999999999,"15min":0.88400000000000001},"min":{"1min":0.64300000000000002,"5min":0.623,"15min":0.53400000000000003},"max":{"1min":1.0029999999999999,"5min":1.2330000000000001,"15min":4.3310000000000004},"last":0.93100000000000005}]},{"osd":4,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.75900000000000001,"5min":0.67300000000000004,"15min":0.376},"max":{"1min":1.0600000000000001,"5min":1.2270000000000001,"15min":4.2270000000000003},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96999999999999997,"15min":0.94099999999999995},"min":{"1min":0.80800000000000005,"5min":0.69599999999999995,"15min":0.55700000000000005},"max":{"1min":1.0840000000000001,"5min":1.671,"15min":1.706},"last":0.90700000000000003}]},{"osd":5,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.89900000000000002,"15min":0.92000000000000004},"min":{"1min":0.71499999999999997,"5min":0.56999999999999995,"15min":0.39000000000000001},"max":{"1min":0.95099999999999996,"5min":2.4449999999999998,"15min":2.5960000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":0.96199999999999997},"min":{"1min":0.74299999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":0.97099999999999997,"5min":1.4890000000000001,"15min":4.7590000000000003},"last":0.84299999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315144,"kb_used_data":9440,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056696,"statfs":{"total":96636764160,"available":96314056704,"internally_reserved":0,"allocated":9666560,"data_stored":5483786,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55600000000000005,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.38100000000000001,"15min":0.29199999999999998},"max":{"1min":0.66700000000000004,"5min":0.82899999999999996,"15min":3.4900000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66800000000000004,"15min":0.68500000000000005},"min":{"1min":0.52800000000000002,"5min":0.442,"15min":0.27500000000000002},"max":{"1min":0.92700000000000005,"5min":1.0209999999999999,"15min":3.1019999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.67400000000000004},"min":{"1min":0.54900000000000004,"5min":0.443,"15min":0.317},"max":{"1min":0.94799999999999995,"5min":0.97299999999999998,"15min":3.5019999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81899999999999995,"15min":0.79000000000000004},"min":{"1min":0.66700000000000004,"5min":0.50700000000000001,"15min":0.41799999999999998},"max":{"1min":1.0620000000000001,"5min":1.0660000000000001,"15min":3.1779999999999999},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.69899999999999995,"5min":0.57899999999999996,"15min":0.40300000000000002},"max":{"1min":1.1120000000000001,"5min":1.2250000000000001,"15min":1.71},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88400000000000001,"15min":0.92100000000000004},"min":{"1min":0.67800000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.0229999999999999,"5min":1.26,"15min":4.3719999999999999},"last":0.70599999999999996}]},{"osd":4,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84599999999999997,"15min":0.876},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.48899999999999999},"max":{"1min":0.91500000000000004,"5min":1.123,"15min":2.4990000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.98899999999999999,"15min":0.93999999999999995},"min":{"1min":0.79500000000000004,"5min":0.70999999999999996,"15min":0.443},"max":{"1min":1.181,"5min":1.262,"15min":1.6679999999999999},"last":0.92300000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.90700000000000003,"15min":0.92400000000000004},"min":{"1min":0.79800000000000004,"5min":0.56100000000000005,"15min":0.504},"max":{"1min":1.0289999999999999,"5min":1.3300000000000001,"15min":2.5840000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.04,"5min":1.0209999999999999,"15min":0.97999999999999998},"min":{"1min":0.84799999999999998,"5min":0.67200000000000004,"15min":0.47399999999999998},"max":{"1min":1.2310000000000001,"5min":1.53,"15min":1.752},"last":1.02}]}]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315076,"kb_used_data":9396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056764,"statfs":{"total":96636764160,"available":96314126336,"internally_reserved":0,"allocated":9621504,"data_stored":4954882,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.191},"max":{"1min":0.69499999999999995,"5min":2.4470000000000001,"15min":3.8849999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.71799999999999997},"min":{"1min":0.54900000000000004,"5min":0.40300000000000002,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65700000000000003,"15min":0.67600000000000005},"min":{"1min":0.53200000000000003,"5min":0.501,"15min":0.23300000000000001},"max":{"1min":0.88100000000000001,"5min":1.3799999999999999,"15min":3.6030000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.871,"15min":0.81699999999999995},"min":{"1min":0.66600000000000004,"5min":0.63100000000000001,"15min":0.30399999999999999},"max":{"1min":1.0569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.89700000000000002}]},{"osd":3,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84899999999999998,"15min":0.84599999999999997},"min":{"1min":0.621,"5min":0.60099999999999998,"15min":0.28100000000000003},"max":{"1min":0.98099999999999998,"5min":2.2440000000000002,"15min":3.7909999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.93799999999999994,"15min":0.92500000000000004},"min":{"1min":0.69799999999999995,"5min":0.65800000000000003,"15min":0.313},"max":{"1min":1.248,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.0089999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.002,"15min":0.95299999999999996},"min":{"1min":0.82199999999999995,"5min":0.57199999999999995,"15min":0.40799999999999997},"max":{"1min":1.085,"5min":2.8530000000000002,"15min":3.6709999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96899999999999997,"15min":0.996},"min":{"1min":0.67900000000000005,"5min":0.67400000000000004,"15min":0.33700000000000002},"max":{"1min":1.175,"5min":2.726,"15min":3.5270000000000001},"last":1.111}]},{"osd":5,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88800000000000001},"min":{"1min":0.80800000000000005,"5min":0.66700000000000004,"15min":0.36799999999999999},"max":{"1min":1.0289999999999999,"5min":1.389,"15min":3.6469999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.98899999999999999,"15min":0.97499999999999998},"min":{"1min":0.71899999999999997,"5min":0.70099999999999996,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":2.7429999999999999,"15min":4.3280000000000003},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:38.961 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T12:03:38.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.061+0000 7f999d200700 10 monclient: get_monmap_and_config 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.061+0000 7f999d200700 10 monclient: build_initial_monmap 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.061+0000 7f999d200700 10 monclient: monmap: 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:39.068267+0000 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:39.068267+0000 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:39.067 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:39.068 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:39.068 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 1 Processor -- start 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 1 -- start start 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 10 monclient: init 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 10 monclient(hunting): picked mon.noname-b con 0x7f99980a2200 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 10 monclient(hunting): picked mon.noname-a con 0x7f99980a01e0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 10 monclient(hunting): picked mon.noname-c con 0x7f9998138600 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999806e890 con 0x7f99980a01e0 2024-09-18T12:03:39.069 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99980a16e0 con 0x7f9998138600 2024-09-18T12:03:39.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99980a25d0 con 0x7f99980a2200 2024-09-18T12:03:39.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:39.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999d200700 10 monclient(hunting): authenticate will time out at 1990.909117s 2024-09-18T12:03:39.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f99980a01e0 0x7f999809b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:52018/0 (socket says 172.21.15.112:52018) 2024-09-18T12:03:39.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999659c700 1 -- 172.21.15.112:0/328892658 learned_addr learned my addr 172.21.15.112:0/328892658 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:39.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9996d9d700 1 --1- 172.21.15.112:0/328892658 >> v1:172.21.15.112:6790/0 conn(0x7f99980a2200 0x7f999809ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35700/0 (socket says 172.21.15.112:35700) 2024-09-18T12:03:39.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9996d9d700 1 -- 172.21.15.112:0/328892658 >> v1:172.21.15.112:6790/0 conn(0x7f99980a2200 legacy=0x7f999809ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999659c700 1 -- 172.21.15.112:0/328892658 >> v1:172.21.15.112:6789/0 conn(0x7f99980a01e0 legacy=0x7f999809b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f999759e700 1 -- 172.21.15.112:0/328892658 >> v1:172.21.15.129:6789/0 conn(0x7f9998138600 legacy=0x7f999813a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152748445 0 0) 0x7f99980a25d0 con 0x7f99980a2200 2024-09-18T12:03:39.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9995d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9995d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9995d9b700 10 monclient(hunting): my global_id is 154247 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9984003570 con 0x7f99980a2200 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295434978 0 0) 0x7f99880017d0 con 0x7f99980a01e0 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.065+0000 7f9995d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient(hunting): my global_id is 164148 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99980a25d0 con 0x7f99980a01e0 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600929712 0 0) 0x7f99980a16e0 con 0x7f9998138600 2024-09-18T12:03:39.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient(hunting): my global_id is 144334 2024-09-18T12:03:39.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99880017d0 con 0x7f9998138600 2024-09-18T12:03:39.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3840115127 0 0) 0x7f99980a25d0 con 0x7f99980a01e0 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 >> v1:172.21.15.112:6790/0 conn(0x7f99980a2200 legacy=0x7f999809ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 >> v1:172.21.15.129:6789/0 conn(0x7f9998138600 legacy=0x7f999813a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: _finish_hunting 0 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 monclient: found mon.noname-a 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999809bc90 con 0x7f99980a01e0 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: _finish_auth 0 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 5 monclient: authenticate success, global_id 164148 2024-09-18T12:03:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9988001d50 con 0x7f99980a01e0 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: dump: 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.076 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9988001f30 con 0x7f99980a01e0 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 4 set_mon_vals no callback set 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 1 -- 172.21.15.112:0/328892658 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9988004c70 con 0x7f99980a01e0 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f9995d9b700 10 monclient: dump: 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.077 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.078 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 10 monclient: shutdown 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 1 -- 172.21.15.112:0/328892658 >> v1:172.21.15.112:6789/0 conn(0x7f99980a01e0 legacy=0x7f999809b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 1 -- 172.21.15.112:0/328892658 shutdown_connections 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 1 -- 172.21.15.112:0/328892658 >> 172.21.15.112:0/328892658 conn(0x7f99980680d0 msgr2=0x7f99980a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 1 -- 172.21.15.112:0/328892658 shutdown_connections 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 1 -- 172.21.15.112:0/328892658 wait complete. 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 10 monclient: build_initial_monmap 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 10 monclient: monmap: 2024-09-18T12:03:39.083 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 1 librados: starting msgr at 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.069+0000 7f999d200700 1 librados: starting objecter 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 1 Processor -- start 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 1 -- start start 2024-09-18T12:03:39.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 1 librados: setting wanted keys 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 1 librados: calling monclient init 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 10 monclient: init 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 10 monclient(hunting): picked mon.noname-c con 0x7f99980a01e0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 10 monclient(hunting): picked mon.noname-b con 0x7f99980a2200 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 10 monclient(hunting): picked mon.noname-a con 0x7f9998138600 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99980a0f80 con 0x7f9998138600 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999809ec90 con 0x7f99980a2200 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99981363a0 con 0x7f99980a01e0 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999d200700 10 monclient(hunting): authenticate will time out at 1990.916903s 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f9996d9d700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f99980a01e0 0x7f9998141b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:35714/0 (socket says 172.21.15.112:35714) 2024-09-18T12:03:39.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999759e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9998138600 0x7f9998149ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:52022/0 (socket says 172.21.15.112:52022) 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f9996d9d700 1 -- 172.21.15.112:0/2656858962 learned_addr learned my addr 172.21.15.112:0/2656858962 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999759e700 1 -- 172.21.15.112:0/2656858962 >> v1:172.21.15.112:6789/0 conn(0x7f9998138600 legacy=0x7f9998149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f9996d9d700 1 -- 172.21.15.112:0/2656858962 >> v1:172.21.15.112:6790/0 conn(0x7f99980a01e0 legacy=0x7f9998141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 client.?.objecter ms_handle_connect 0x7f9998138600 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 client.?.objecter ms_handle_connect 0x7f99980a01e0 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f999659c700 1 -- 172.21.15.112:0/2656858962 >> v1:172.21.15.129:6789/0 conn(0x7f99980a2200 legacy=0x7f99981467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 client.?.objecter ms_handle_connect 0x7f99980a2200 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279999231 0 0) 0x7f99980a0f80 con 0x7f9998138600 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): my global_id is 164151 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9978004f10 con 0x7f9998138600 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008936281 0 0) 0x7f99981363a0 con 0x7f99980a01e0 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): my global_id is 154250 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99980a0f80 con 0x7f99980a01e0 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280717199 0 0) 0x7f999809ec90 con 0x7f99980a2200 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 10 monclient(hunting): my global_id is 144337 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.073+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99981363a0 con 0x7f99980a2200 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1116944152 0 0) 0x7f9978004f10 con 0x7f9998138600 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f999809ec90 con 0x7f9998138600 2024-09-18T12:03:39.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9980001c40 con 0x7f9998138600 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient(hunting): dump: 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279948625 0 0) 0x7f99980a0f80 con 0x7f99980a01e0 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9978004f10 con 0x7f99980a01e0 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f99900019b0 con 0x7f99980a01e0 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient(hunting): dump: 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.089 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3852422796 0 0) 0x7f999809ec90 con 0x7f9998138600 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 >> v1:172.21.15.112:6790/0 conn(0x7f99980a01e0 legacy=0x7f9998141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 >> v1:172.21.15.129:6789/0 conn(0x7f99980a2200 legacy=0x7f99981467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient: _finish_hunting 0 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 monclient: found mon.a 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99981379c0 con 0x7f9998138600 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient: _finish_auth 0 2024-09-18T12:03:39.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2132713182 0 0) 0x7f9978004f10 con 0x7f99980a01e0 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 5 monclient: authenticate success, global_id 164151 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 10 monclient: _renew_subs 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9998142280 con 0x7f9998138600 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 10 client.164151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 10 monclient: _renew_subs 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99800022b0 con 0x7f9998138600 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9998130a30 con 0x7f9998138600 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 1 librados: init done 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:39.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f9980002600 con 0x7f9998138600 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999559a700 4 set_mon_vals no callback set 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 monclient: dump: 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.092 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999813a130 con 0x7f9998138600 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f9980002890 con 0x7f9998138600 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f9980043b00 con 0x7f9998138600 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 10 client.164151.objecter ms_dispatch 0x7f9998000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 3 client.164151.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 3 client.164151.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f998effd700 20 client.164151.objecter dump_active .. 0 homeless 2024-09-18T12:03:39.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.077+0000 7f9996d9d700 10 monclient: get_auth_request con 0x7f9978038480 auth_method 0 2024-09-18T12:03:39.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.081+0000 7f998effd700 10 client.164151.objecter ms_handle_connect 0x7f9978038480 2024-09-18T12:03:39.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.081+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f9980002d20 con 0x7f9998138600 2024-09-18T12:03:39.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.081+0000 7f998effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:39.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.081+0000 7f998effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:39.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.245+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f999805f2e0 con 0x7f9978038480 2024-09-18T12:03:39.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f998effd700 1 -- 172.21.15.112:0/2656858962 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134033 (unknown 2965378022 0 767431492) 0x7f999805f2e0 con 0x7f9978038480 2024-09-18T12:03:39.259 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T12:03:39.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 10 librados: watch_flush enter 2024-09-18T12:03:39.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 10 librados: watch_flush exit 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 20 client.164151.objecter shutdown clearing up homeless session... 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 10 client.164151.objecter successfully canceled tick 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 >> v1:172.21.15.129:6812/20806 conn(0x7f9978038480 legacy=0x7f997803a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 10 monclient: shutdown 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 >> v1:172.21.15.112:6789/0 conn(0x7f9998138600 legacy=0x7f9998149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 shutdown_connections 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 >> 172.21.15.112:0/2656858962 conn(0x7f99980680d0 msgr2=0x7f999809da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 shutdown_connections 2024-09-18T12:03:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 1 -- 172.21.15.112:0/2656858962 wait complete. 2024-09-18T12:03:39.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.257+0000 7f999d200700 1 librados: shutdown 2024-09-18T12:03:39.263 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T12:03:39.275 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1274,"stamp":"2024-09-18T12:03:38.380870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1898908,"kb_used_data":56444,"kb_used_omap":9,"kb_used_meta":1842294,"kb_avail":564332132,"statfs":{"total":579820584960,"available":577876103168,"internally_reserved":0,"allocated":57798656,"data_stored":30786034,"data_compressed":320902,"data_compressed_allocated":876544,"data_compressed_original":2744320,"omap_allocated":10224,"internal_metadata":1886509072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003849"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522453+0000","last_change":"2024-09-18T11:45:25.374991+0000","last_active":"2024-09-18T12:02:11.522453+0000","last_peered":"2024-09-18T12:02:11.522453+0000","last_clean":"2024-09-18T12:02:11.522453+0000","last_became_active":"2024-09-18T11:45:25.373642+0000","last_became_peered":"2024-09-18T11:45:25.373642+0000","last_unstale":"2024-09-18T12:02:11.522453+0000","last_undegraded":"2024-09-18T12:02:11.522453+0000","last_fullsized":"2024-09-18T12:02:11.522453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:57:36.040393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519515+0000","last_change":"2024-09-18T11:45:25.329845+0000","last_active":"2024-09-18T12:02:11.519515+0000","last_peered":"2024-09-18T12:02:11.519515+0000","last_clean":"2024-09-18T12:02:11.519515+0000","last_became_active":"2024-09-18T11:45:25.328032+0000","last_became_peered":"2024-09-18T11:45:25.328032+0000","last_unstale":"2024-09-18T12:02:11.519515+0000","last_undegraded":"2024-09-18T12:02:11.519515+0000","last_fullsized":"2024-09-18T12:02:11.519515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:13.680321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913257+0000","last_change":"2024-09-18T11:45:25.306434+0000","last_active":"2024-09-18T12:02:10.913257+0000","last_peered":"2024-09-18T12:02:10.913257+0000","last_clean":"2024-09-18T12:02:10.913257+0000","last_became_active":"2024-09-18T11:45:25.305872+0000","last_became_peered":"2024-09-18T11:45:25.305872+0000","last_unstale":"2024-09-18T12:02:10.913257+0000","last_undegraded":"2024-09-18T12:02:10.913257+0000","last_fullsized":"2024-09-18T12:02:10.913257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:30:00.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519584+0000","last_change":"2024-09-18T11:45:25.411469+0000","last_active":"2024-09-18T12:02:11.519584+0000","last_peered":"2024-09-18T12:02:11.519584+0000","last_clean":"2024-09-18T12:02:11.519584+0000","last_became_active":"2024-09-18T11:45:25.411133+0000","last_became_peered":"2024-09-18T11:45:25.411133+0000","last_unstale":"2024-09-18T12:02:11.519584+0000","last_undegraded":"2024-09-18T12:02:11.519584+0000","last_fullsized":"2024-09-18T12:02:11.519584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:02:10.494289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523123+0000","last_change":"2024-09-18T11:45:25.400321+0000","last_active":"2024-09-18T12:02:11.523123+0000","last_peered":"2024-09-18T12:02:11.523123+0000","last_clean":"2024-09-18T12:02:11.523123+0000","last_became_active":"2024-09-18T11:45:25.399898+0000","last_became_peered":"2024-09-18T11:45:25.399898+0000","last_unstale":"2024-09-18T12:02:11.523123+0000","last_undegraded":"2024-09-18T12:02:11.523123+0000","last_fullsized":"2024-09-18T12:02:11.523123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:14.033638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.520050+0000","last_change":"2024-09-18T11:45:25.397202+0000","last_active":"2024-09-18T12:02:11.520050+0000","last_peered":"2024-09-18T12:02:11.520050+0000","last_clean":"2024-09-18T12:02:11.520050+0000","last_became_active":"2024-09-18T11:45:25.396975+0000","last_became_peered":"2024-09-18T11:45:25.396975+0000","last_unstale":"2024-09-18T12:02:11.520050+0000","last_undegraded":"2024-09-18T12:02:11.520050+0000","last_fullsized":"2024-09-18T12:02:11.520050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:37:37.634734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.910998+0000","last_change":"2024-09-18T11:45:25.341093+0000","last_active":"2024-09-18T12:02:10.910998+0000","last_peered":"2024-09-18T12:02:10.910998+0000","last_clean":"2024-09-18T12:02:10.910998+0000","last_became_active":"2024-09-18T11:45:25.340726+0000","last_became_peered":"2024-09-18T11:45:25.340726+0000","last_unstale":"2024-09-18T12:02:10.910998+0000","last_undegraded":"2024-09-18T12:02:10.910998+0000","last_fullsized":"2024-09-18T12:02:10.910998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:36:59.172751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960537+0000","last_change":"2024-09-18T11:45:25.408737+0000","last_active":"2024-09-18T12:02:10.960537+0000","last_peered":"2024-09-18T12:02:10.960537+0000","last_clean":"2024-09-18T12:02:10.960537+0000","last_became_active":"2024-09-18T11:45:25.408464+0000","last_became_peered":"2024-09-18T11:45:25.408464+0000","last_unstale":"2024-09-18T12:02:10.960537+0000","last_undegraded":"2024-09-18T12:02:10.960537+0000","last_fullsized":"2024-09-18T12:02:10.960537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:44:57.076431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.909867+0000","last_change":"2024-09-18T11:45:25.344531+0000","last_active":"2024-09-18T12:02:10.909867+0000","last_peered":"2024-09-18T12:02:10.909867+0000","last_clean":"2024-09-18T12:02:10.909867+0000","last_became_active":"2024-09-18T11:45:25.343097+0000","last_became_peered":"2024-09-18T11:45:25.343097+0000","last_unstale":"2024-09-18T12:02:10.909867+0000","last_undegraded":"2024-09-18T12:02:10.909867+0000","last_fullsized":"2024-09-18T12:02:10.909867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:49.438992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519649+0000","last_change":"2024-09-18T11:45:25.377493+0000","last_active":"2024-09-18T12:02:11.519649+0000","last_peered":"2024-09-18T12:02:11.519649+0000","last_clean":"2024-09-18T12:02:11.519649+0000","last_became_active":"2024-09-18T11:45:25.377194+0000","last_became_peered":"2024-09-18T11:45:25.377194+0000","last_unstale":"2024-09-18T12:02:11.519649+0000","last_undegraded":"2024-09-18T12:02:11.519649+0000","last_fullsized":"2024-09-18T12:02:11.519649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:44:53.107224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522770+0000","last_change":"2024-09-18T11:45:25.346404+0000","last_active":"2024-09-18T12:02:11.522770+0000","last_peered":"2024-09-18T12:02:11.522770+0000","last_clean":"2024-09-18T12:02:11.522770+0000","last_became_active":"2024-09-18T11:45:25.341682+0000","last_became_peered":"2024-09-18T11:45:25.341682+0000","last_unstale":"2024-09-18T12:02:11.522770+0000","last_undegraded":"2024-09-18T12:02:11.522770+0000","last_fullsized":"2024-09-18T12:02:11.522770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:04.010096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522588+0000","last_change":"2024-09-18T11:45:25.383080+0000","last_active":"2024-09-18T12:02:11.522588+0000","last_peered":"2024-09-18T12:02:11.522588+0000","last_clean":"2024-09-18T12:02:11.522588+0000","last_became_active":"2024-09-18T11:45:25.377663+0000","last_became_peered":"2024-09-18T11:45:25.377663+0000","last_unstale":"2024-09-18T12:02:11.522588+0000","last_undegraded":"2024-09-18T12:02:11.522588+0000","last_fullsized":"2024-09-18T12:02:11.522588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:46:12.443901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":649,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.914509+0000","last_change":"2024-09-18T11:44:24.299141+0000","last_active":"2024-09-18T12:02:10.914509+0000","last_peered":"2024-09-18T12:02:10.914509+0000","last_clean":"2024-09-18T12:02:10.914509+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T12:02:10.914509+0000","last_undegraded":"2024-09-18T12:02:10.914509+0000","last_fullsized":"2024-09-18T12:02:10.914509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:35:52.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960533+0000","last_change":"2024-09-18T11:45:25.405730+0000","last_active":"2024-09-18T12:02:10.960533+0000","last_peered":"2024-09-18T12:02:10.960533+0000","last_clean":"2024-09-18T12:02:10.960533+0000","last_became_active":"2024-09-18T11:45:25.405515+0000","last_became_peered":"2024-09-18T11:45:25.405515+0000","last_unstale":"2024-09-18T12:02:10.960533+0000","last_undegraded":"2024-09-18T12:02:10.960533+0000","last_fullsized":"2024-09-18T12:02:10.960533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:39:09.362580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911684+0000","last_change":"2024-09-18T11:44:24.208317+0000","last_active":"2024-09-18T12:02:10.911684+0000","last_peered":"2024-09-18T12:02:10.911684+0000","last_clean":"2024-09-18T12:02:10.911684+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T12:02:10.911684+0000","last_undegraded":"2024-09-18T12:02:10.911684+0000","last_fullsized":"2024-09-18T12:02:10.911684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:54.979951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913034+0000","last_change":"2024-09-18T11:45:25.306008+0000","last_active":"2024-09-18T12:02:10.913034+0000","last_peered":"2024-09-18T12:02:10.913034+0000","last_clean":"2024-09-18T12:02:10.913034+0000","last_became_active":"2024-09-18T11:45:25.305686+0000","last_became_peered":"2024-09-18T11:45:25.305686+0000","last_unstale":"2024-09-18T12:02:10.913034+0000","last_undegraded":"2024-09-18T12:02:10.913034+0000","last_fullsized":"2024-09-18T12:02:10.913034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:31:41.349569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519991+0000","last_change":"2024-09-18T11:44:24.213224+0000","last_active":"2024-09-18T12:02:11.519991+0000","last_peered":"2024-09-18T12:02:11.519991+0000","last_clean":"2024-09-18T12:02:11.519991+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T12:02:11.519991+0000","last_undegraded":"2024-09-18T12:02:11.519991+0000","last_fullsized":"2024-09-18T12:02:11.519991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:47.310939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"20'7","reported_seq":660,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.910044+0000","last_change":"2024-09-18T11:45:25.403798+0000","last_active":"2024-09-18T12:02:10.910044+0000","last_peered":"2024-09-18T12:02:10.910044+0000","last_clean":"2024-09-18T12:02:10.910044+0000","last_became_active":"2024-09-18T11:45:25.403586+0000","last_became_peered":"2024-09-18T11:45:25.403586+0000","last_unstale":"2024-09-18T12:02:10.910044+0000","last_undegraded":"2024-09-18T12:02:10.910044+0000","last_fullsized":"2024-09-18T12:02:10.910044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:53:41.121574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":654,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522879+0000","last_change":"2024-09-18T11:44:24.507933+0000","last_active":"2024-09-18T12:02:11.522879+0000","last_peered":"2024-09-18T12:02:11.522879+0000","last_clean":"2024-09-18T12:02:11.522879+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T12:02:11.522879+0000","last_undegraded":"2024-09-18T12:02:11.522879+0000","last_fullsized":"2024-09-18T12:02:11.522879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:56.697013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522889+0000","last_change":"2024-09-18T11:45:25.369556+0000","last_active":"2024-09-18T12:02:11.522889+0000","last_peered":"2024-09-18T12:02:11.522889+0000","last_clean":"2024-09-18T12:02:11.522889+0000","last_became_active":"2024-09-18T11:45:25.367362+0000","last_became_peered":"2024-09-18T11:45:25.367362+0000","last_unstale":"2024-09-18T12:02:11.522889+0000","last_undegraded":"2024-09-18T12:02:11.522889+0000","last_fullsized":"2024-09-18T12:02:11.522889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:23:17.811113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":743,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.907687+0000","last_change":"2024-09-18T11:44:21.097910+0000","last_active":"2024-09-18T12:02:10.907687+0000","last_peered":"2024-09-18T12:02:10.907687+0000","last_clean":"2024-09-18T12:02:10.907687+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T12:02:10.907687+0000","last_undegraded":"2024-09-18T12:02:10.907687+0000","last_fullsized":"2024-09-18T12:02:10.907687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:48.394073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":655,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522634+0000","last_change":"2024-09-18T11:44:24.508079+0000","last_active":"2024-09-18T12:02:11.522634+0000","last_peered":"2024-09-18T12:02:11.522634+0000","last_clean":"2024-09-18T12:02:11.522634+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T12:02:11.522634+0000","last_undegraded":"2024-09-18T12:02:11.522634+0000","last_fullsized":"2024-09-18T12:02:11.522634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:41:30.648862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913627+0000","last_change":"2024-09-18T11:45:25.407247+0000","last_active":"2024-09-18T12:02:10.913627+0000","last_peered":"2024-09-18T12:02:10.913627+0000","last_clean":"2024-09-18T12:02:10.913627+0000","last_became_active":"2024-09-18T11:45:25.406430+0000","last_became_peered":"2024-09-18T11:45:25.406430+0000","last_unstale":"2024-09-18T12:02:10.913627+0000","last_undegraded":"2024-09-18T12:02:10.913627+0000","last_fullsized":"2024-09-18T12:02:10.913627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:56:03.017626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"17'2","reported_seq":660,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519718+0000","last_change":"2024-09-18T11:44:24.214482+0000","last_active":"2024-09-18T12:02:11.519718+0000","last_peered":"2024-09-18T12:02:11.519718+0000","last_clean":"2024-09-18T12:02:11.519718+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T12:02:11.519718+0000","last_undegraded":"2024-09-18T12:02:11.519718+0000","last_fullsized":"2024-09-18T12:02:11.519718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:26:34.055867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523238+0000","last_change":"2024-09-18T11:45:25.365786+0000","last_active":"2024-09-18T12:02:11.523238+0000","last_peered":"2024-09-18T12:02:11.523238+0000","last_clean":"2024-09-18T12:02:11.523238+0000","last_became_active":"2024-09-18T11:45:25.363235+0000","last_became_peered":"2024-09-18T11:45:25.363235+0000","last_unstale":"2024-09-18T12:02:11.523238+0000","last_undegraded":"2024-09-18T12:02:11.523238+0000","last_fullsized":"2024-09-18T12:02:11.523238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:15:20.066742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":649,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.912589+0000","last_change":"2024-09-18T11:44:24.298975+0000","last_active":"2024-09-18T12:02:10.912589+0000","last_peered":"2024-09-18T12:02:10.912589+0000","last_clean":"2024-09-18T12:02:10.912589+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T12:02:10.912589+0000","last_undegraded":"2024-09-18T12:02:10.912589+0000","last_fullsized":"2024-09-18T12:02:10.912589+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:27.175495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960440+0000","last_change":"2024-09-18T11:45:25.356461+0000","last_active":"2024-09-18T12:02:10.960440+0000","last_peered":"2024-09-18T12:02:10.960440+0000","last_clean":"2024-09-18T12:02:10.960440+0000","last_became_active":"2024-09-18T11:45:25.356178+0000","last_became_peered":"2024-09-18T11:45:25.356178+0000","last_unstale":"2024-09-18T12:02:10.960440+0000","last_undegraded":"2024-09-18T12:02:10.960440+0000","last_fullsized":"2024-09-18T12:02:10.960440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:06:26.698280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.908011+0000","last_change":"2024-09-18T11:44:24.208252+0000","last_active":"2024-09-18T12:02:10.908011+0000","last_peered":"2024-09-18T12:02:10.908011+0000","last_clean":"2024-09-18T12:02:10.908011+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T12:02:10.908011+0000","last_undegraded":"2024-09-18T12:02:10.908011+0000","last_fullsized":"2024-09-18T12:02:10.908011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:08:27.257954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.520056+0000","last_change":"2024-09-18T11:45:25.396966+0000","last_active":"2024-09-18T12:02:11.520056+0000","last_peered":"2024-09-18T12:02:11.520056+0000","last_clean":"2024-09-18T12:02:11.520056+0000","last_became_active":"2024-09-18T11:45:25.396428+0000","last_became_peered":"2024-09-18T11:45:25.396428+0000","last_unstale":"2024-09-18T12:02:11.520056+0000","last_undegraded":"2024-09-18T12:02:11.520056+0000","last_fullsized":"2024-09-18T12:02:11.520056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:08:22.504624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911592+0000","last_change":"2024-09-18T11:45:25.399933+0000","last_active":"2024-09-18T12:02:10.911592+0000","last_peered":"2024-09-18T12:02:10.911592+0000","last_clean":"2024-09-18T12:02:10.911592+0000","last_became_active":"2024-09-18T11:45:25.399695+0000","last_became_peered":"2024-09-18T11:45:25.399695+0000","last_unstale":"2024-09-18T12:02:10.911592+0000","last_undegraded":"2024-09-18T12:02:10.911592+0000","last_fullsized":"2024-09-18T12:02:10.911592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:01:15.698698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913437+0000","last_change":"2024-09-18T11:45:25.387334+0000","last_active":"2024-09-18T12:02:10.913437+0000","last_peered":"2024-09-18T12:02:10.913437+0000","last_clean":"2024-09-18T12:02:10.913437+0000","last_became_active":"2024-09-18T11:45:25.386932+0000","last_became_peered":"2024-09-18T11:45:25.386932+0000","last_unstale":"2024-09-18T12:02:10.913437+0000","last_undegraded":"2024-09-18T12:02:10.913437+0000","last_fullsized":"2024-09-18T12:02:10.913437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:34:23.587779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522768+0000","last_change":"2024-09-18T11:45:25.403356+0000","last_active":"2024-09-18T12:02:11.522768+0000","last_peered":"2024-09-18T12:02:11.522768+0000","last_clean":"2024-09-18T12:02:11.522768+0000","last_became_active":"2024-09-18T11:45:25.401979+0000","last_became_peered":"2024-09-18T11:45:25.401979+0000","last_unstale":"2024-09-18T12:02:11.522768+0000","last_undegraded":"2024-09-18T12:02:11.522768+0000","last_fullsized":"2024-09-18T12:02:11.522768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:29:14.433201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523103+0000","last_change":"2024-09-18T11:45:25.381976+0000","last_active":"2024-09-18T12:02:11.523103+0000","last_peered":"2024-09-18T12:02:11.523103+0000","last_clean":"2024-09-18T12:02:11.523103+0000","last_became_active":"2024-09-18T11:45:25.381652+0000","last_became_peered":"2024-09-18T11:45:25.381652+0000","last_unstale":"2024-09-18T12:02:11.523103+0000","last_undegraded":"2024-09-18T12:02:11.523103+0000","last_fullsized":"2024-09-18T12:02:11.523103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:21.420291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522937+0000","last_change":"2024-09-18T11:45:25.381748+0000","last_active":"2024-09-18T12:02:11.522937+0000","last_peered":"2024-09-18T12:02:11.522937+0000","last_clean":"2024-09-18T12:02:11.522937+0000","last_became_active":"2024-09-18T11:45:25.379657+0000","last_became_peered":"2024-09-18T11:45:25.379657+0000","last_unstale":"2024-09-18T12:02:11.522937+0000","last_undegraded":"2024-09-18T12:02:11.522937+0000","last_fullsized":"2024-09-18T12:02:11.522937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:44:38.923827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523379+0000","last_change":"2024-09-18T11:45:25.340204+0000","last_active":"2024-09-18T12:02:11.523379+0000","last_peered":"2024-09-18T12:02:11.523379+0000","last_clean":"2024-09-18T12:02:11.523379+0000","last_became_active":"2024-09-18T11:45:25.338772+0000","last_became_peered":"2024-09-18T11:45:25.338772+0000","last_unstale":"2024-09-18T12:02:11.523379+0000","last_undegraded":"2024-09-18T12:02:11.523379+0000","last_fullsized":"2024-09-18T12:02:11.523379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:09.409329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522520+0000","last_change":"2024-09-18T11:45:25.369783+0000","last_active":"2024-09-18T12:02:11.522520+0000","last_peered":"2024-09-18T12:02:11.522520+0000","last_clean":"2024-09-18T12:02:11.522520+0000","last_became_active":"2024-09-18T11:45:25.369294+0000","last_became_peered":"2024-09-18T11:45:25.369294+0000","last_unstale":"2024-09-18T12:02:11.522520+0000","last_undegraded":"2024-09-18T12:02:11.522520+0000","last_fullsized":"2024-09-18T12:02:11.522520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:20:30.771482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.912743+0000","last_change":"2024-09-18T11:45:25.393624+0000","last_active":"2024-09-18T12:02:10.912743+0000","last_peered":"2024-09-18T12:02:10.912743+0000","last_clean":"2024-09-18T12:02:10.912743+0000","last_became_active":"2024-09-18T11:45:25.393093+0000","last_became_peered":"2024-09-18T11:45:25.393093+0000","last_unstale":"2024-09-18T12:02:10.912743+0000","last_undegraded":"2024-09-18T12:02:10.912743+0000","last_fullsized":"2024-09-18T12:02:10.912743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:17:30.506236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911484+0000","last_change":"2024-09-18T11:45:25.382303+0000","last_active":"2024-09-18T12:02:10.911484+0000","last_peered":"2024-09-18T12:02:10.911484+0000","last_clean":"2024-09-18T12:02:10.911484+0000","last_became_active":"2024-09-18T11:45:25.380362+0000","last_became_peered":"2024-09-18T11:45:25.380362+0000","last_unstale":"2024-09-18T12:02:10.911484+0000","last_undegraded":"2024-09-18T12:02:10.911484+0000","last_fullsized":"2024-09-18T12:02:10.911484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:11:31.985540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523224+0000","last_change":"2024-09-18T11:45:25.358090+0000","last_active":"2024-09-18T12:02:11.523224+0000","last_peered":"2024-09-18T12:02:11.523224+0000","last_clean":"2024-09-18T12:02:11.523224+0000","last_became_active":"2024-09-18T11:45:25.357477+0000","last_became_peered":"2024-09-18T11:45:25.357477+0000","last_unstale":"2024-09-18T12:02:11.523224+0000","last_undegraded":"2024-09-18T12:02:11.523224+0000","last_fullsized":"2024-09-18T12:02:11.523224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:54:53.792263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960338+0000","last_change":"2024-09-18T11:45:25.394821+0000","last_active":"2024-09-18T12:02:10.960338+0000","last_peered":"2024-09-18T12:02:10.960338+0000","last_clean":"2024-09-18T12:02:10.960338+0000","last_became_active":"2024-09-18T11:45:25.394579+0000","last_became_peered":"2024-09-18T11:45:25.394579+0000","last_unstale":"2024-09-18T12:02:10.960338+0000","last_undegraded":"2024-09-18T12:02:10.960338+0000","last_fullsized":"2024-09-18T12:02:10.960338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:59.330658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.914779+0000","last_change":"2024-09-18T11:45:25.405243+0000","last_active":"2024-09-18T12:02:10.914779+0000","last_peered":"2024-09-18T12:02:10.914779+0000","last_clean":"2024-09-18T12:02:10.914779+0000","last_became_active":"2024-09-18T11:45:25.403069+0000","last_became_peered":"2024-09-18T11:45:25.403069+0000","last_unstale":"2024-09-18T12:02:10.914779+0000","last_undegraded":"2024-09-18T12:02:10.914779+0000","last_fullsized":"2024-09-18T12:02:10.914779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:50:14.772996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607788,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315068,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056772,"statfs":{"total":96636764160,"available":96314134528,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.876,"15min":0.873},"min":{"1min":0.56799999999999995,"5min":0.48499999999999999,"15min":0.41499999999999998},"max":{"1min":1.1140000000000001,"5min":1.244,"15min":1.53},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.85199999999999998},"min":{"1min":0.70499999999999996,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0129999999999999,"5min":1.294,"15min":1.454},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85599999999999998,"15min":0.879},"min":{"1min":0.58299999999999996,"5min":0.52900000000000003,"15min":0.46500000000000002},"max":{"1min":0.999,"5min":1.2529999999999999,"15min":1.325},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.85999999999999999,"15min":0.86199999999999999},"min":{"1min":0.73799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.98899999999999999,"5min":1.208,"15min":1.2689999999999999},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79400000000000004,"15min":0.82199999999999995},"min":{"1min":0.70799999999999996,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":0.95699999999999996,"5min":1.2410000000000001,"15min":2.8900000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89000000000000001,"15min":0.86899999999999999},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.439},"max":{"1min":1.1060000000000001,"5min":1.6060000000000001,"15min":3.0760000000000001},"last":0.93200000000000005}]},{"osd":3,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.79100000000000004,"15min":0.77200000000000002},"min":{"1min":0.61199999999999999,"5min":0.45800000000000002,"15min":0.372},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.6950000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.56799999999999995,"5min":0.439,"15min":0.35299999999999998},"max":{"1min":1.198,"5min":1.204,"15min":1.446},"last":0.77200000000000002}]},{"osd":4,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.82799999999999996,"15min":0.86099999999999999},"min":{"1min":0.72899999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.159,"5min":1.159,"15min":1.323},"last":0.876},{"interface":"front","average":{"1min":0.995,"5min":0.98099999999999998,"15min":0.95999999999999996},"min":{"1min":0.82099999999999995,"5min":0.54600000000000004,"15min":0.439},"max":{"1min":1.232,"5min":1.3089999999999999,"15min":1.4670000000000001},"last":0.66800000000000004}]}]},{"osd":4,"up_from":12,"seq":51539607787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319244,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052596,"statfs":{"total":96636764160,"available":96309858304,"internally_reserved":0,"allocated":9613312,"data_stored":4953859,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.83699999999999997,"15min":0.82899999999999996},"min":{"1min":0.64800000000000002,"5min":0.51800000000000002,"15min":0.47699999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.3120000000000001},"last":1.04},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83399999999999996,"15min":0.83099999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.46400000000000002},"max":{"1min":1.131,"5min":1.272,"15min":1.296},"last":0.94199999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.504},"max":{"1min":0.92200000000000004,"5min":1.3169999999999999,"15min":1.331},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90500000000000003,"15min":0.85799999999999998},"min":{"1min":0.69199999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3600000000000001},"last":0.96099999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78800000000000003,"15min":0.82399999999999995},"min":{"1min":0.627,"5min":0.56799999999999995,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":1.0329999999999999,"15min":1.294},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.92800000000000005,"15min":0.90300000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.432},"max":{"1min":1.206,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.74299999999999999}]},{"osd":3,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.57999999999999996,"5min":0.48899999999999999,"15min":0.47499999999999998},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.879,"15min":0.89300000000000002},"min":{"1min":0.70599999999999996,"5min":0.5,"15min":0.44600000000000001},"max":{"1min":1.0900000000000001,"5min":1.4019999999999999,"15min":1.496},"last":0.82799999999999996}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.56999999999999995,"5min":0.53500000000000003,"15min":0.40100000000000002},"max":{"1min":1.085,"5min":1.4430000000000001,"15min":2.899},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0289999999999999,"15min":0.97199999999999998},"min":{"1min":0.76900000000000002,"5min":0.69299999999999995,"15min":0.56599999999999995},"max":{"1min":1.28,"5min":1.466,"15min":3.004},"last":0.995}]}]},{"osd":3,"up_from":12,"seq":51539607787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319292,"kb_used_data":9444,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052548,"statfs":{"total":96636764160,"available":96309809152,"internally_reserved":0,"allocated":9670656,"data_stored":5483791,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75,"15min":0.80500000000000005},"min":{"1min":0.55600000000000005,"5min":0.5,"15min":0.35999999999999999},"max":{"1min":1.2350000000000001,"5min":1.331,"15min":5.1390000000000002},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.82099999999999995,"15min":0.876},"min":{"1min":0.66700000000000004,"5min":0.56799999999999995,"15min":0.40999999999999998},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":7.0449999999999999},"last":0.58499999999999996}]},{"osd":1,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.79200000000000004,"5min":0.59699999999999998,"15min":0.505},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":7.4880000000000004},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93400000000000005,"15min":0.91500000000000004},"min":{"1min":0.69799999999999995,"5min":0.55000000000000004,"15min":0.45800000000000002},"max":{"1min":0.97999999999999998,"5min":1.325,"15min":4.9699999999999998},"last":0.88800000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.626,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":0.95499999999999996,"5min":1.419,"15min":5.2080000000000002},"last":0.626},{"interface":"front","average":{"1min":0.876,"5min":0.91900000000000004,"15min":0.96599999999999997},"min":{"1min":0.67600000000000005,"5min":0.63600000000000001,"15min":0.441},"max":{"1min":1.1259999999999999,"5min":1.3280000000000001,"15min":7.4180000000000001},"last":0.85099999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.73099999999999998},"min":{"1min":0.53400000000000003,"5min":0.49299999999999999,"15min":0.39600000000000002},"max":{"1min":0.94199999999999995,"5min":1.2569999999999999,"15min":4.7169999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97699999999999998},"min":{"1min":0.70499999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.361,"5min":1.7150000000000001,"15min":4.6260000000000003},"last":0.91800000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.82399999999999995,"15min":0.86499999999999999},"min":{"1min":0.621,"5min":0.54500000000000004,"15min":0.34699999999999998},"max":{"1min":1.159,"5min":1.208,"15min":7.4470000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0740000000000001,"15min":1.1160000000000001},"min":{"1min":0.871,"5min":0.68600000000000005,"15min":0.63200000000000001},"max":{"1min":1.4390000000000001,"5min":1.581,"15min":7.6269999999999998},"last":0.97499999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607787,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315084,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056756,"statfs":{"total":96636764160,"available":96314118144,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58499999999999996},"min":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.33100000000000002},"max":{"1min":0.98499999999999999,"5min":1.212,"15min":1.212},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72499999999999998},"min":{"1min":0.60999999999999999,"5min":0.48199999999999998,"15min":0.28699999999999998},"max":{"1min":1.02,"5min":1.254,"15min":1.254},"last":0.73899999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.495,"5min":0.41599999999999998,"15min":0.316},"max":{"1min":0.76100000000000001,"5min":0.93500000000000005,"15min":3.0899999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.56999999999999995,"5min":0.49299999999999999,"15min":0.41999999999999998},"max":{"1min":0.84199999999999997,"5min":1.577,"15min":4.1699999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.86599999999999999,"15min":0.89100000000000001},"min":{"1min":0.67800000000000005,"5min":0.52500000000000002,"15min":0.36899999999999999},"max":{"1min":1.0680000000000001,"5min":1.3799999999999999,"15min":2.6389999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.86199999999999999,"15min":0.88400000000000001},"min":{"1min":0.64300000000000002,"5min":0.623,"15min":0.53400000000000003},"max":{"1min":1.0029999999999999,"5min":1.2330000000000001,"15min":4.3310000000000004},"last":0.93100000000000005}]},{"osd":4,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.75900000000000001,"5min":0.67300000000000004,"15min":0.376},"max":{"1min":1.0600000000000001,"5min":1.2270000000000001,"15min":4.2270000000000003},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96999999999999997,"15min":0.94099999999999995},"min":{"1min":0.80800000000000005,"5min":0.69599999999999995,"15min":0.55700000000000005},"max":{"1min":1.0840000000000001,"5min":1.671,"15min":1.706},"last":0.90700000000000003}]},{"osd":5,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.89900000000000002,"15min":0.92000000000000004},"min":{"1min":0.71499999999999997,"5min":0.56999999999999995,"15min":0.39000000000000001},"max":{"1min":0.95099999999999996,"5min":2.4449999999999998,"15min":2.5960000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":0.96199999999999997},"min":{"1min":0.74299999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":0.97099999999999997,"5min":1.4890000000000001,"15min":4.7590000000000003},"last":0.84299999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315144,"kb_used_data":9440,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056696,"statfs":{"total":96636764160,"available":96314056704,"internally_reserved":0,"allocated":9666560,"data_stored":5483786,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55600000000000005,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.38100000000000001,"15min":0.29199999999999998},"max":{"1min":0.66700000000000004,"5min":0.82899999999999996,"15min":3.4900000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66800000000000004,"15min":0.68500000000000005},"min":{"1min":0.52800000000000002,"5min":0.442,"15min":0.27500000000000002},"max":{"1min":0.92700000000000005,"5min":1.0209999999999999,"15min":3.1019999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.67400000000000004},"min":{"1min":0.54900000000000004,"5min":0.443,"15min":0.317},"max":{"1min":0.94799999999999995,"5min":0.97299999999999998,"15min":3.5019999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81899999999999995,"15min":0.79000000000000004},"min":{"1min":0.66700000000000004,"5min":0.50700000000000001,"15min":0.41799999999999998},"max":{"1min":1.0620000000000001,"5min":1.0660000000000001,"15min":3.1779999999999999},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.69899999999999995,"5min":0.57899999999999996,"15min":0.40300000000000002},"max":{"1min":1.1120000000000001,"5min":1.2250000000000001,"15min":1.71},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88400000000000001,"15min":0.92100000000000004},"min":{"1min":0.67800000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.0229999999999999,"5min":1.26,"15min":4.3719999999999999},"last":0.70599999999999996}]},{"osd":4,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84599999999999997,"15min":0.876},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.48899999999999999},"max":{"1min":0.91500000000000004,"5min":1.123,"15min":2.4990000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.98899999999999999,"15min":0.93999999999999995},"min":{"1min":0.79500000000000004,"5min":0.70999999999999996,"15min":0.443},"max":{"1min":1.181,"5min":1.262,"15min":1.6679999999999999},"last":0.92300000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.90700000000000003,"15min":0.92400000000000004},"min":{"1min":0.79800000000000004,"5min":0.56100000000000005,"15min":0.504},"max":{"1min":1.0289999999999999,"5min":1.3300000000000001,"15min":2.5840000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.04,"5min":1.0209999999999999,"15min":0.97999999999999998},"min":{"1min":0.84799999999999998,"5min":0.67200000000000004,"15min":0.47399999999999998},"max":{"1min":1.2310000000000001,"5min":1.53,"15min":1.752},"last":1.02}]}]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315076,"kb_used_data":9396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056764,"statfs":{"total":96636764160,"available":96314126336,"internally_reserved":0,"allocated":9621504,"data_stored":4954882,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.191},"max":{"1min":0.69499999999999995,"5min":2.4470000000000001,"15min":3.8849999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.71799999999999997},"min":{"1min":0.54900000000000004,"5min":0.40300000000000002,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65700000000000003,"15min":0.67600000000000005},"min":{"1min":0.53200000000000003,"5min":0.501,"15min":0.23300000000000001},"max":{"1min":0.88100000000000001,"5min":1.3799999999999999,"15min":3.6030000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.871,"15min":0.81699999999999995},"min":{"1min":0.66600000000000004,"5min":0.63100000000000001,"15min":0.30399999999999999},"max":{"1min":1.0569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.89700000000000002}]},{"osd":3,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84899999999999998,"15min":0.84599999999999997},"min":{"1min":0.621,"5min":0.60099999999999998,"15min":0.28100000000000003},"max":{"1min":0.98099999999999998,"5min":2.2440000000000002,"15min":3.7909999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.93799999999999994,"15min":0.92500000000000004},"min":{"1min":0.69799999999999995,"5min":0.65800000000000003,"15min":0.313},"max":{"1min":1.248,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.0089999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.002,"15min":0.95299999999999996},"min":{"1min":0.82199999999999995,"5min":0.57199999999999995,"15min":0.40799999999999997},"max":{"1min":1.085,"5min":2.8530000000000002,"15min":3.6709999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96899999999999997,"15min":0.996},"min":{"1min":0.67900000000000005,"5min":0.67400000000000004,"15min":0.33700000000000002},"max":{"1min":1.175,"5min":2.726,"15min":3.5270000000000001},"last":1.111}]},{"osd":5,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88800000000000001},"min":{"1min":0.80800000000000005,"5min":0.66700000000000004,"15min":0.36799999999999999},"max":{"1min":1.0289999999999999,"5min":1.389,"15min":3.6469999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.98899999999999999,"15min":0.97499999999999998},"min":{"1min":0.71899999999999997,"5min":0.70099999999999996,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":2.7429999999999999,"15min":4.3280000000000003},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:39.278 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:03:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.381+0000 7feeb2c3b700 10 monclient: get_monmap_and_config 2024-09-18T12:03:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.381+0000 7feeb2c3b700 10 monclient: build_initial_monmap 2024-09-18T12:03:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.381+0000 7feeb2c3b700 10 monclient: monmap: 2024-09-18T12:03:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:39.387646+0000 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:39.387646+0000 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:39.387 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.381+0000 7feeb2c3b700 1 Processor -- start 2024-09-18T12:03:39.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.381+0000 7feeb2c3b700 1 -- start start 2024-09-18T12:03:39.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.381+0000 7feeb2c3b700 10 monclient: init 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 10 monclient(hunting): picked mon.noname-b con 0x7feeac0a2040 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 10 monclient(hunting): picked mon.noname-c con 0x7feeac12edc0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 10 monclient(hunting): picked mon.noname-a con 0x7feeac12f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0981a0 con 0x7feeac12f8a0 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac06c120 con 0x7feeac12edc0 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac06eef0 con 0x7feeac0a2040 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 10 monclient(hunting): authenticate will time out at 1991.228569s 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb243a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7feeac12f8a0 0x7feeac137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46028/0 (socket says 172.21.15.112:46028) 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb243a700 1 -- 172.21.15.112:0/2951959176 learned_addr learned my addr 172.21.15.112:0/2951959176 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:39.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb1c39700 1 --1- 172.21.15.112:0/2951959176 >> v1:172.21.15.112:6790/0 conn(0x7feeac0a2040 0x7feeac097a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58672/0 (socket says 172.21.15.112:58672) 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb243a700 1 -- 172.21.15.112:0/2951959176 >> v1:172.21.15.112:6789/0 conn(0x7feeac12f8a0 legacy=0x7feeac137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb1c39700 1 -- 172.21.15.112:0/2951959176 >> v1:172.21.15.112:6790/0 conn(0x7feeac0a2040 legacy=0x7feeac097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb1438700 1 -- 172.21.15.112:0/2951959176 >> v1:172.21.15.129:6789/0 conn(0x7feeac12edc0 legacy=0x7feeac12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930478788 0 0) 0x7feeac0981a0 con 0x7feeac12f8a0 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): my global_id is 164154 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee98003570 con 0x7feeac12f8a0 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320743566 0 0) 0x7feeac06eef0 con 0x7feeac0a2040 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): my global_id is 154253 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeac0981a0 con 0x7feeac0a2040 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662478529 0 0) 0x7feeac06c120 con 0x7feeac12edc0 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient(hunting): my global_id is 144340 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeac06eef0 con 0x7feeac12edc0 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3956714214 0 0) 0x7fee98003570 con 0x7feeac12f8a0 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 >> v1:172.21.15.112:6790/0 conn(0x7feeac0a2040 legacy=0x7feeac097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 >> v1:172.21.15.129:6789/0 conn(0x7feeac12edc0 legacy=0x7feeac12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient: _finish_hunting 0 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 monclient: found mon.noname-a 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:39.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac096dd0 con 0x7feeac12f8a0 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient: _finish_auth 0 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7feea4001cc0 con 0x7feeac12f8a0 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 5 monclient: authenticate success, global_id 164154 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb2c3b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.385+0000 7feeb0c37700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb0c37700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb0c37700 10 monclient: dump: 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.395 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feea4001f50 con 0x7feeac12f8a0 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb0c37700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 4 set_mon_vals no callback set 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb0c37700 1 -- 172.21.15.112:0/2951959176 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7feea4004e10 con 0x7feeac12f8a0 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb0c37700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb0c37700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:39.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb0c37700 10 monclient: dump: 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 10 monclient: shutdown 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:39.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 -- 172.21.15.112:0/2951959176 >> v1:172.21.15.112:6789/0 conn(0x7feeac12f8a0 legacy=0x7feeac137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 -- 172.21.15.112:0/2951959176 shutdown_connections 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 -- 172.21.15.112:0/2951959176 >> 172.21.15.112:0/2951959176 conn(0x7feeac067da0 msgr2=0x7feeac0a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 -- 172.21.15.112:0/2951959176 shutdown_connections 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 -- 172.21.15.112:0/2951959176 wait complete. 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 10 monclient: build_initial_monmap 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 10 monclient: monmap: 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:39.398 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:39.399 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:39.399 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 librados: starting msgr at 2024-09-18T12:03:39.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 librados: starting objecter 2024-09-18T12:03:39.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 Processor -- start 2024-09-18T12:03:39.403 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 -- start start 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 librados: setting wanted keys 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 1 librados: calling monclient init 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 10 monclient: init 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.389+0000 7feeb2c3b700 10 monclient(hunting): picked mon.noname-b con 0x7feeac0a2040 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb2c3b700 10 monclient(hunting): picked mon.noname-c con 0x7feeac12edc0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb2c3b700 10 monclient(hunting): picked mon.noname-a con 0x7feeac12f8a0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb2c3b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac119f90 con 0x7feeac12f8a0 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb2c3b700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac11c3b0 con 0x7feeac0a2040 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb2c3b700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac11c0d0 con 0x7feeac12edc0 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb2c3b700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:39.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb2c3b700 10 monclient(hunting): authenticate will time out at 1991.236156s 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb243a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7feeac12f8a0 0x7feeac145440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46038/0 (socket says 172.21.15.112:46038) 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb243a700 1 -- 172.21.15.112:0/3052976068 learned_addr learned my addr 172.21.15.112:0/3052976068 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb243a700 1 -- 172.21.15.112:0/3052976068 >> v1:172.21.15.112:6789/0 conn(0x7feeac12f8a0 legacy=0x7feeac145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 client.?.objecter ms_handle_connect 0x7feeac12f8a0 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 client.?.objecter ms_handle_connect 0x7feeac12edc0 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb1438700 1 -- 172.21.15.112:0/3052976068 >> v1:172.21.15.112:6790/0 conn(0x7feeac12edc0 legacy=0x7feeac13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 client.?.objecter ms_handle_connect 0x7feeac0a2040 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb1c39700 1 -- 172.21.15.112:0/3052976068 >> v1:172.21.15.129:6789/0 conn(0x7feeac0a2040 legacy=0x7feeac12dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feeb243a700 1 --1- 172.21.15.112:0/3052976068 >> v1:172.21.15.112:6789/0 conn(0x7feeac12f8a0 0x7feeac145440 :-1 s=OPENED pgs=718 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7feeac119f90 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429393067 0 0) 0x7feeac11c0d0 con 0x7feeac12edc0 2024-09-18T12:03:39.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): my global_id is 154256 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee90004f10 con 0x7feeac12edc0 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092104621 0 0) 0x7feeac11c3b0 con 0x7feeac0a2040 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): my global_id is 144343 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeac11c0d0 con 0x7feeac0a2040 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1051208804 0 0) 0x7fee90004f10 con 0x7feeac12edc0 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feeac11c3b0 con 0x7feeac12edc0 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7feea8001c00 con 0x7feeac12edc0 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): dump: 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064848298 0 0) 0x7feeac11c0d0 con 0x7feeac0a2040 2024-09-18T12:03:39.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee90004f10 con 0x7feeac0a2040 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fee9c001b70 con 0x7feeac0a2040 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 10 monclient(hunting): dump: 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.408 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.393+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2202642774 0 0) 0x7feeac11c3b0 con 0x7feeac12edc0 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 >> v1:172.21.15.129:6789/0 conn(0x7feeac0a2040 legacy=0x7feeac12dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 >> v1:172.21.15.112:6789/0 conn(0x7feeac12f8a0 legacy=0x7feeac145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 monclient: found mon.c 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac11c690 con 0x7feeac12edc0 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 monclient: _finish_auth 0 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 5 monclient: authenticate success, global_id 154256 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 10 monclient: _renew_subs 2024-09-18T12:03:39.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773869622 0 0) 0x7feeac119f90 con 0x7feeac12f8a0 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac12e6b0 con 0x7feeac12edc0 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 10 client.154256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 10 monclient: _renew_subs 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac13d9a0 con 0x7feeac12edc0 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 1 librados: init done 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac09bad0 con 0x7feeac12edc0 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feea80022b0 con 0x7feeac12edc0 2024-09-18T12:03:39.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7feea8002600 con 0x7feeac12edc0 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 monclient: dump: 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:39.411 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:39.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea37fe700 4 set_mon_vals no callback set 2024-09-18T12:03:39.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7feea80027e0 con 0x7feeac12edc0 2024-09-18T12:03:39.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7feea8043c00 con 0x7feeac12edc0 2024-09-18T12:03:39.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 10 client.154256.objecter ms_dispatch 0x7feeac000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:39.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 3 client.154256.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:39.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 3 client.154256.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:39.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feea1ffb700 20 client.154256.objecter dump_active .. 0 homeless 2024-09-18T12:03:39.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.397+0000 7feeb1c39700 10 monclient: get_auth_request con 0x7fee90038380 auth_method 0 2024-09-18T12:03:39.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.401+0000 7feea1ffb700 10 client.154256.objecter ms_handle_connect 0x7fee90038380 2024-09-18T12:03:39.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.405+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7feea8014640 con 0x7feeac12edc0 2024-09-18T12:03:39.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.405+0000 7feea1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:39.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.405+0000 7feea1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:39.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.569+0000 7feeb2c3b700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:03:39.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.569+0000 7feeb2c3b700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:03:39.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.569+0000 7feeb2c3b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:39.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.569+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feeac05f2e0 con 0x7feeac12edc0 2024-09-18T12:03:39.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.569+0000 7feea1ffb700 1 -- 172.21.15.112:0/3052976068 <== mon.2 v1:172.21.15.112:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v699) v1 ==== 74+0+12741 (unknown 3608138097 0 2795684330) 0x7feea8014460 con 0x7feeac12edc0 2024-09-18T12:03:39.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.569+0000 7feea1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:03:39.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.569+0000 7feea1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:03:39.574 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T12:03:39.574 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":699,"fsid":"e68e10b8-0874-4c25-87a0-f38d9ca2e0d2","created":"2024-09-18T11:44:09.317663+0000","modified":"2024-09-18T12:02:11.898642+0000","last_up_change":"2024-09-18T11:44:19.034653+0000","last_in_change":"2024-09-18T11:44:11.615001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":104,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:44:19.901214+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-18T11:44:20.029398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-18T11:45:24.012859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"ecb0abea-5208-453b-86e2-58b64b0634ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":20984}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":20984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":20984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":20984}]},"public_addr":"172.21.15.112:6804/20984","cluster_addr":"172.21.15.112:6805/20984","heartbeat_back_addr":"172.21.15.112:6807/20984","heartbeat_front_addr":"172.21.15.112:6806/20984","state":["exists","up"]},{"osd":1,"uuid":"3164789a-58af-4cf7-bd86-260d70dd02c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":20985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":20985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":20985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":20985}]},"public_addr":"172.21.15.112:6800/20985","cluster_addr":"172.21.15.112:6801/20985","heartbeat_back_addr":"172.21.15.112:6803/20985","heartbeat_front_addr":"172.21.15.112:6802/20985","state":["exists","up"]},{"osd":2,"uuid":"0223fabc-71f3-4642-8818-d99dbd5ba501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":20986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":20986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":20986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":20986}]},"public_addr":"172.21.15.112:6808/20986","cluster_addr":"172.21.15.112:6809/20986","heartbeat_back_addr":"172.21.15.112:6811/20986","heartbeat_front_addr":"172.21.15.112:6810/20986","state":["exists","up"]},{"osd":3,"uuid":"07461389-250c-4e30-8890-4267bddeac4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6804","nonce":21049}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6805","nonce":21049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6807","nonce":21049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6806","nonce":21049}]},"public_addr":"172.21.15.129:6804/21049","cluster_addr":"172.21.15.129:6805/21049","heartbeat_back_addr":"172.21.15.129:6807/21049","heartbeat_front_addr":"172.21.15.129:6806/21049","state":["exists","up"]},{"osd":4,"uuid":"ad3a24d7-0278-4137-a499-e7b7337e1698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6808","nonce":21047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6809","nonce":21047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6811","nonce":21047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6810","nonce":21047}]},"public_addr":"172.21.15.129:6808/21047","cluster_addr":"172.21.15.129:6809/21047","heartbeat_back_addr":"172.21.15.129:6811/21047","heartbeat_front_addr":"172.21.15.129:6810/21047","state":["exists","up"]},{"osd":5,"uuid":"174b7f7f-23bc-45c9-9245-7c3edc4b6235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6800","nonce":21048}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6801","nonce":21048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6803","nonce":21048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.129:6802","nonce":21048}]},"public_addr":"172.21.15.129:6800/21048","cluster_addr":"172.21.15.129:6801/21048","heartbeat_back_addr":"172.21.15.129:6803/21048","heartbeat_front_addr":"172.21.15.129:6802/21048","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.965457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:16.813648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.196234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.055175+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.022483+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:44:17.092059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":13,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":54,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":74,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":82,"snaps":[{"begin":2,"length":1}]},{"pool":90,"snaps":[{"begin":2,"length":1},{"begin":6,"length":2}]},{"pool":92,"snaps":[{"begin":4,"length":4}]},{"pool":103,"snaps":[{"begin":2,"length":1}]},{"pool":109,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":119,"snaps":[{"begin":12,"length":1}]},{"pool":125,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":144,"snaps":[{"begin":14,"length":1}]},{"pool":155,"snaps":[{"begin":17,"length":1}]},{"pool":161,"snaps":[{"begin":19,"length":2}]},{"pool":176,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":5,"length":1}]},{"pool":244,"snaps":[{"begin":7,"length":1}]},{"pool":248,"snaps":[{"begin":10,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:03:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 10 librados: watch_flush enter 2024-09-18T12:03:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 10 librados: watch_flush exit 2024-09-18T12:03:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 20 client.154256.objecter shutdown clearing up homeless session... 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 10 client.154256.objecter successfully canceled tick 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 >> v1:172.21.15.129:6812/20806 conn(0x7fee90038380 legacy=0x7fee9003a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 10 monclient: shutdown 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 >> v1:172.21.15.112:6790/0 conn(0x7feeac12edc0 legacy=0x7feeac13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 shutdown_connections 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 >> 172.21.15.112:0/3052976068 conn(0x7feeac067da0 msgr2=0x7feeac09f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 shutdown_connections 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 1 -- 172.21.15.112:0/3052976068 wait complete. 2024-09-18T12:03:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:39.573+0000 7feeb2c3b700 1 librados: shutdown 2024-09-18T12:03:40.593 INFO:tasks.ceph:Scrubbing osd.0 2024-09-18T12:03:40.594 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:03:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.705+0000 7f6e76af7700 10 monclient: get_monmap_and_config 2024-09-18T12:03:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.705+0000 7f6e76af7700 10 monclient: build_initial_monmap 2024-09-18T12:03:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.705+0000 7f6e76af7700 10 monclient: monmap: 2024-09-18T12:03:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:40.711501+0000 2024-09-18T12:03:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:40.711501+0000 2024-09-18T12:03:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:40.712 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.705+0000 7f6e76af7700 1 Processor -- start 2024-09-18T12:03:40.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.705+0000 7f6e76af7700 1 -- start start 2024-09-18T12:03:40.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.705+0000 7f6e76af7700 10 monclient: init 2024-09-18T12:03:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:03:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 10 monclient(hunting): picked mon.noname-b con 0x7f6e700a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 10 monclient(hunting): picked mon.noname-a con 0x7f6e700a0e60 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 10 monclient(hunting): picked mon.noname-c con 0x7f6e70097b70 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e70097f90 con 0x7f6e700a0e60 2024-09-18T12:03:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e700a1550 con 0x7f6e70097b70 2024-09-18T12:03:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e70096dd0 con 0x7f6e700a0380 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 10 monclient(hunting): authenticate will time out at 1992.552819s 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e75af5700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f6e700a0380 0x7f6e700a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58686/0 (socket says 172.21.15.112:58686) 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e75af5700 1 -- 172.21.15.112:0/240544774 learned_addr learned my addr 172.21.15.112:0/240544774 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e752f4700 1 --1- 172.21.15.112:0/240544774 >> v1:172.21.15.112:6789/0 conn(0x7f6e700a0e60 0x7f6e70137690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46054/0 (socket says 172.21.15.112:46054) 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e752f4700 1 -- 172.21.15.112:0/240544774 >> v1:172.21.15.112:6789/0 conn(0x7f6e700a0e60 legacy=0x7f6e70137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e75af5700 1 -- 172.21.15.112:0/240544774 >> v1:172.21.15.112:6790/0 conn(0x7f6e700a0380 legacy=0x7f6e700a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764270091 0 0) 0x7f6e70097f90 con 0x7f6e700a0e60 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient(hunting): my global_id is 164160 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5c003570 con 0x7f6e700a0e60 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e762f6700 1 -- 172.21.15.112:0/240544774 >> v1:172.21.15.129:6789/0 conn(0x7f6e70097b70 legacy=0x7f6e7013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450748818 0 0) 0x7f6e70096dd0 con 0x7f6e700a0380 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient(hunting): my global_id is 154259 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e70097f90 con 0x7f6e700a0380 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 597119381 0 0) 0x7f6e5c003570 con 0x7f6e700a0e60 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 >> v1:172.21.15.112:6790/0 conn(0x7f6e700a0380 legacy=0x7f6e700a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 >> v1:172.21.15.129:6789/0 conn(0x7f6e70097b70 legacy=0x7f6e7013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient: _finish_hunting 0 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 monclient: found mon.noname-a 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e700a1e40 con 0x7f6e700a0e60 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient: _finish_auth 0 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 5 monclient: authenticate success, global_id 164160 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e60001940 con 0x7f6e700a0e60 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e76af7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.709+0000 7f6e74af3700 10 monclient: dump: 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e60001bd0 con 0x7f6e700a0e60 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e74af3700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e74af3700 1 -- 172.21.15.112:0/240544774 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e600049d0 con 0x7f6e700a0e60 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e74af3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 4 set_mon_vals no callback set 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e74af3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e74af3700 10 monclient: dump: 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient: shutdown 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- 172.21.15.112:0/240544774 >> v1:172.21.15.112:6789/0 conn(0x7f6e700a0e60 legacy=0x7f6e70137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- 172.21.15.112:0/240544774 shutdown_connections 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- 172.21.15.112:0/240544774 >> 172.21.15.112:0/240544774 conn(0x7f6e70067680 msgr2=0x7f6e70067a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- 172.21.15.112:0/240544774 shutdown_connections 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- 172.21.15.112:0/240544774 wait complete. 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient: build_initial_monmap 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient: monmap: 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:40.725 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 librados: starting msgr at 2024-09-18T12:03:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 librados: starting objecter 2024-09-18T12:03:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 Processor -- start 2024-09-18T12:03:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- start start 2024-09-18T12:03:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 librados: setting wanted keys 2024-09-18T12:03:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 librados: calling monclient init 2024-09-18T12:03:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient: init 2024-09-18T12:03:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:03:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient(hunting): picked mon.noname-c con 0x7f6e70097b70 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient(hunting): picked mon.noname-a con 0x7f6e700a0380 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient(hunting): picked mon.noname-b con 0x7f6e700a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e70120d90 con 0x7f6e700a0380 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7011e340 con 0x7f6e700a0e60 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e75af5700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f6e70097b70 0x7f6e7011d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58702/0 (socket says 172.21.15.112:58702) 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7011f490 con 0x7f6e70097b70 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e75af5700 1 -- 172.21.15.112:0/4216906889 learned_addr learned my addr 172.21.15.112:0/4216906889 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.713+0000 7f6e76af7700 10 monclient(hunting): authenticate will time out at 1992.559852s 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e752f4700 1 -- 172.21.15.112:0/4216906889 >> v1:172.21.15.112:6789/0 conn(0x7f6e700a0380 legacy=0x7f6e7011dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 client.?.objecter ms_handle_connect 0x7f6e700a0380 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 client.?.objecter ms_handle_connect 0x7f6e70097b70 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e75af5700 1 -- 172.21.15.112:0/4216906889 >> v1:172.21.15.112:6790/0 conn(0x7f6e70097b70 legacy=0x7f6e7011d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e762f6700 1 -- 172.21.15.112:0/4216906889 >> v1:172.21.15.129:6789/0 conn(0x7f6e700a0e60 legacy=0x7f6e7014a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275441637 0 0) 0x7f6e70120d90 con 0x7f6e700a0380 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): my global_id is 164163 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e54004f10 con 0x7f6e700a0380 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 client.?.objecter ms_handle_connect 0x7f6e700a0e60 2024-09-18T12:03:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228873768 0 0) 0x7f6e7011f490 con 0x7f6e70097b70 2024-09-18T12:03:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): my global_id is 154262 2024-09-18T12:03:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e70120d90 con 0x7f6e70097b70 2024-09-18T12:03:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423252239 0 0) 0x7f6e7011e340 con 0x7f6e700a0e60 2024-09-18T12:03:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): my global_id is 144349 2024-09-18T12:03:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e7011f490 con 0x7f6e700a0e60 2024-09-18T12:03:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2358091873 0 0) 0x7f6e54004f10 con 0x7f6e700a0380 2024-09-18T12:03:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e7011e340 con 0x7f6e700a0380 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e60004830 con 0x7f6e700a0380 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): dump: 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3048082906 0 0) 0x7f6e70120d90 con 0x7f6e70097b70 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e54004f10 con 0x7f6e70097b70 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e68002030 con 0x7f6e70097b70 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient(hunting): dump: 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 852827046 0 0) 0x7f6e7011e340 con 0x7f6e700a0380 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 >> v1:172.21.15.112:6790/0 conn(0x7f6e70097b70 legacy=0x7f6e7011d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 >> v1:172.21.15.129:6789/0 conn(0x7f6e700a0e60 legacy=0x7f6e7014a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 monclient: found mon.a 2024-09-18T12:03:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7011ef80 con 0x7f6e700a0380 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient: _finish_auth 0 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 5 monclient: authenticate success, global_id 164163 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 10 monclient: _renew_subs 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70120ab0 con 0x7f6e700a0380 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 10 client.164163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 10 monclient: _renew_subs 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7011fb50 con 0x7f6e700a0380 2024-09-18T12:03:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e76af7700 1 librados: init done 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e60001580 con 0x7f6e700a0380 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6e60005120 con 0x7f6e700a0380 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e65ffb700 10 monclient: dump: 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.717+0000 7f6e677fe700 4 set_mon_vals no callback set 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f6e6000f9a0 con 0x7f6e700a0380 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f6e60043d30 con 0x7f6e700a0380 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e65ffb700 10 client.164163.objecter ms_dispatch 0x7f6e70000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e65ffb700 3 client.164163.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e65ffb700 3 client.164163.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e65ffb700 20 client.164163.objecter dump_active .. 0 homeless 2024-09-18T12:03:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 10 client.164163.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 15 client.164163.objecter _session_command_op_assign -1 1 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 20 client.164163.objecter _get_session s=0x7f6e50002880 osd=0 3 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 20 client.164163.objecter put_session s=0x7f6e50002880 osd=0 3 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 20 client.164163.objecter _get_session s=0x7f6e50002880 osd=0 3 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 15 client.164163.objecter _session_command_op_remove -1 1 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 20 client.164163.objecter get_session s=0x7f6e50002880 osd=0 3 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 15 client.164163.objecter _session_command_op_assign 0 1 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 20 client.164163.objecter put_session s=0x7f6e50002880 osd=0 4 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 10 client.164163.objecter _send_command 1 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6804/20984 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e50006cd0 con 0x7f6e50002f70 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e75af5700 10 monclient: get_auth_request con 0x7f6e540381a0 auth_method 0 2024-09-18T12:03:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.721+0000 7f6e762f6700 10 monclient: get_auth_request con 0x7f6e50002f70 auth_method 0 2024-09-18T12:03:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.725+0000 7f6e65ffb700 10 client.164163.objecter ms_handle_connect 0x7f6e540381a0 2024-09-18T12:03:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.725+0000 7f6e65ffb700 10 client.164163.objecter ms_handle_connect 0x7f6e50002f70 2024-09-18T12:03:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.725+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== osd.0 v1:172.21.15.112:6804/20984 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f6e50006cd0 con 0x7f6e50002f70 2024-09-18T12:03:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.725+0000 7f6e65ffb700 10 client.164163.objecter ms_dispatch 0x7f6e70000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T12:03:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.725+0000 7f6e65ffb700 10 client.164163.objecter _finish_command 1 = system:0 2024-09-18T12:03:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.725+0000 7f6e65ffb700 20 client.164163.objecter put_session s=0x7f6e50002880 osd=0 4 2024-09-18T12:03:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.725+0000 7f6e65ffb700 15 client.164163.objecter _session_command_op_remove 0 1 2024-09-18T12:03:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 10 client.164163.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-18T12:03:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 15 client.164163.objecter _session_command_op_assign -1 2 2024-09-18T12:03:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 20 client.164163.objecter _get_session s=0x7f6e50002880 osd=0 3 2024-09-18T12:03:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 20 client.164163.objecter put_session s=0x7f6e50002880 osd=0 3 2024-09-18T12:03:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 20 client.164163.objecter _get_session s=0x7f6e50002880 osd=0 3 2024-09-18T12:03:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 15 client.164163.objecter _session_command_op_remove -1 2 2024-09-18T12:03:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 20 client.164163.objecter get_session s=0x7f6e50002880 osd=0 3 2024-09-18T12:03:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 15 client.164163.objecter _session_command_op_assign 0 2 2024-09-18T12:03:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 20 client.164163.objecter put_session s=0x7f6e50002880 osd=0 4 2024-09-18T12:03:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 10 client.164163.objecter _send_command 2 2024-09-18T12:03:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 --> v1:172.21.15.112:6804/20984 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6e500011e0 con 0x7f6e50002f70 2024-09-18T12:03:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e65ffb700 1 -- 172.21.15.112:0/4216906889 <== osd.0 v1:172.21.15.112:6804/20984 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f6e500011e0 con 0x7f6e50002f70 2024-09-18T12:03:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e65ffb700 10 client.164163.objecter ms_dispatch 0x7f6e70000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T12:03:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e65ffb700 10 client.164163.objecter _finish_command 2 = system:0 2024-09-18T12:03:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e65ffb700 20 client.164163.objecter put_session s=0x7f6e50002880 osd=0 4 2024-09-18T12:03:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e65ffb700 15 client.164163.objecter _session_command_op_remove 0 2 2024-09-18T12:03:40.749 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:40.749 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:03:40.749 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:40.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 10 librados: watch_flush enter 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 10 librados: watch_flush exit 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 10 client.164163.objecter close_session for osd.0 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 >> v1:172.21.15.112:6804/20984 conn(0x7f6e50002f70 legacy=0x7f6e50005420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 20 client.164163.objecter put_session s=0x7f6e50002880 osd=0 1 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 20 client.164163.objecter shutdown clearing up homeless session... 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 10 client.164163.objecter successfully canceled tick 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 >> v1:172.21.15.129:6812/20806 conn(0x7f6e540381a0 legacy=0x7f6e5403a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 10 monclient: shutdown 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 >> v1:172.21.15.112:6789/0 conn(0x7f6e700a0380 legacy=0x7f6e7011dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e762f6700 1 -- 172.21.15.112:0/4216906889 reap_dead start 2024-09-18T12:03:40.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 shutdown_connections 2024-09-18T12:03:40.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 >> 172.21.15.112:0/4216906889 conn(0x7f6e70067680 msgr2=0x7f6e700a4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:40.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 shutdown_connections 2024-09-18T12:03:40.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 -- 172.21.15.112:0/4216906889 wait complete. 2024-09-18T12:03:40.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.745+0000 7f6e76af7700 1 librados: shutdown 2024-09-18T12:03:40.760 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-18T12:03:40.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.865+0000 7f573629c700 10 monclient: get_monmap_and_config 2024-09-18T12:03:40.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.865+0000 7f573629c700 10 monclient: build_initial_monmap 2024-09-18T12:03:40.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.865+0000 7f573629c700 10 monclient: monmap: 2024-09-18T12:03:40.870 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:40.870 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.870 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:40.871287+0000 2024-09-18T12:03:40.871 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:40.871287+0000 2024-09-18T12:03:40.871 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:40.871 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.871 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:40.871 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:40.871 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:40.871 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 1 Processor -- start 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 1 -- start start 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 10 monclient: init 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 10 monclient(hunting): picked mon.noname-c con 0x7f57300a0380 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 10 monclient(hunting): picked mon.noname-a con 0x7f57300a0750 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 10 monclient(hunting): picked mon.noname-b con 0x7f57301385b0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5730097270 con 0x7f57300a0750 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57300a3170 con 0x7f57300a0380 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5730096cb0 con 0x7f57301385b0 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:40.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573629c700 10 monclient(hunting): authenticate will time out at 1992.713238s 2024-09-18T12:03:40.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f5734a99700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f57300a0750 0x7f57300a0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46078/0 (socket says 172.21.15.112:46078) 2024-09-18T12:03:40.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f5735a9b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f57301385b0 0x7f573013a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58714/0 (socket says 172.21.15.112:58714) 2024-09-18T12:03:40.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f5734a99700 1 -- 172.21.15.112:0/2165955121 learned_addr learned my addr 172.21.15.112:0/2165955121 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:40.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f5734a99700 1 -- 172.21.15.112:0/2165955121 >> v1:172.21.15.112:6789/0 conn(0x7f57300a0750 legacy=0x7f57300a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f5735a9b700 1 -- 172.21.15.112:0/2165955121 >> v1:172.21.15.112:6790/0 conn(0x7f57301385b0 legacy=0x7f573013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.869+0000 7f573529a700 1 -- 172.21.15.112:0/2165955121 >> v1:172.21.15.129:6789/0 conn(0x7f57300a0380 legacy=0x7f5730097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835796013 0 0) 0x7f5730096cb0 con 0x7f57301385b0 2024-09-18T12:03:40.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): my global_id is 154265 2024-09-18T12:03:40.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c003570 con 0x7f57301385b0 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541395790 0 0) 0x7f5730097270 con 0x7f57300a0750 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): my global_id is 164166 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5730096cb0 con 0x7f57300a0750 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26975119 0 0) 0x7f57300a3170 con 0x7f57300a0380 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient(hunting): my global_id is 144352 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5730097270 con 0x7f57300a0380 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704893171 0 0) 0x7f571c003570 con 0x7f57301385b0 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 >> v1:172.21.15.129:6789/0 conn(0x7f57300a0380 legacy=0x7f5730097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 >> v1:172.21.15.112:6789/0 conn(0x7f57300a0750 legacy=0x7f57300a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: _finish_hunting 0 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 monclient: found mon.noname-b 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57300a1420 con 0x7f57301385b0 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: _finish_auth 0 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 5 monclient: authenticate success, global_id 154265 2024-09-18T12:03:40.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5728001d80 con 0x7f57301385b0 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: dump: 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.880 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5728001f60 con 0x7f57301385b0 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 1 -- 172.21.15.112:0/2165955121 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5728004d50 con 0x7f57301385b0 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 4 set_mon_vals no callback set 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f5727fff700 10 monclient: dump: 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.881 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 10 monclient: shutdown 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 -- 172.21.15.112:0/2165955121 >> v1:172.21.15.112:6790/0 conn(0x7f57301385b0 legacy=0x7f573013a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 -- 172.21.15.112:0/2165955121 shutdown_connections 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 -- 172.21.15.112:0/2165955121 >> 172.21.15.112:0/2165955121 conn(0x7f57300a5990 msgr2=0x7f57300a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 -- 172.21.15.112:0/2165955121 shutdown_connections 2024-09-18T12:03:40.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 -- 172.21.15.112:0/2165955121 wait complete. 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 10 monclient: build_initial_monmap 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 10 monclient: monmap: 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:40.888 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 librados: starting msgr at 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 librados: starting objecter 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 Processor -- start 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 -- start start 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 librados: setting wanted keys 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 1 librados: calling monclient init 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.873+0000 7f573629c700 10 monclient: init 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 10 monclient(hunting): picked mon.noname-b con 0x7f57300a0380 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 10 monclient(hunting): picked mon.noname-c con 0x7f57300a0750 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 10 monclient(hunting): picked mon.noname-a con 0x7f57301385b0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:40.889 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57300a2010 con 0x7f57301385b0 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5730138aa0 con 0x7f57300a0380 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57301368a0 con 0x7f57300a0750 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573629c700 10 monclient(hunting): authenticate will time out at 1992.720630s 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f5734a99700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f57300a0750 0x7f5730136190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58724/0 (socket says 172.21.15.112:58724) 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f5734a99700 1 -- 172.21.15.112:0/1565746112 learned_addr learned my addr 172.21.15.112:0/1565746112 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f5735a9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f57301385b0 0x7f573014a1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46090/0 (socket says 172.21.15.112:46090) 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f5734a99700 1 -- 172.21.15.112:0/1565746112 >> v1:172.21.15.112:6790/0 conn(0x7f57300a0750 legacy=0x7f5730136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 client.?.objecter ms_handle_connect 0x7f57300a0750 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 client.?.objecter ms_handle_connect 0x7f57301385b0 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f5735a9b700 1 -- 172.21.15.112:0/1565746112 >> v1:172.21.15.112:6789/0 conn(0x7f57301385b0 legacy=0x7f573014a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 client.?.objecter ms_handle_connect 0x7f57300a0380 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f573529a700 1 -- 172.21.15.112:0/1565746112 >> v1:172.21.15.129:6789/0 conn(0x7f57300a0380 legacy=0x7f5730135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415910405 0 0) 0x7f57301368a0 con 0x7f57300a0750 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): my global_id is 154268 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5714004f10 con 0x7f57300a0750 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034026672 0 0) 0x7f57300a2010 con 0x7f57301385b0 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): my global_id is 164169 2024-09-18T12:03:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57301368a0 con 0x7f57301385b0 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840248114 0 0) 0x7f5730138aa0 con 0x7f57300a0380 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 10 monclient(hunting): my global_id is 144355 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57300a2010 con 0x7f57300a0380 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.877+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1333637613 0 0) 0x7f5714004f10 con 0x7f57300a0750 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5730138aa0 con 0x7f57300a0750 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f572c001c20 con 0x7f57300a0750 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient(hunting): dump: 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 247176233 0 0) 0x7f57301368a0 con 0x7f57301385b0 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5714004f10 con 0x7f57301385b0 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f57280016e0 con 0x7f57301385b0 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient(hunting): dump: 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2278216275 0 0) 0x7f5730138aa0 con 0x7f57300a0750 2024-09-18T12:03:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 >> v1:172.21.15.129:6789/0 conn(0x7f57300a0380 legacy=0x7f5730135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 >> v1:172.21.15.112:6789/0 conn(0x7f57301385b0 legacy=0x7f573014a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient: _finish_hunting 0 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 monclient: found mon.c 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57300983d0 con 0x7f57300a0750 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient: _finish_auth 0 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2139418832 0 0) 0x7f5714004f10 con 0x7f57301385b0 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 5 monclient: authenticate success, global_id 154268 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 10 monclient: _renew_subs 2024-09-18T12:03:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57301374e0 con 0x7f57300a0750 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 10 client.154268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 10 monclient: _renew_subs 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f573009d920 con 0x7f57300a0750 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 1 librados: init done 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f572c004be0 con 0x7f57300a0750 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f572c004f70 con 0x7f57300a0750 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57277fe700 4 set_mon_vals no callback set 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 monclient: dump: 2024-09-18T12:03:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f572c0050d0 con 0x7f57300a0750 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57100050b0 con 0x7f57300a0750 2024-09-18T12:03:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f572c043bd0 con 0x7f57300a0750 2024-09-18T12:03:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 10 client.154268.objecter ms_dispatch 0x7f5730000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 3 client.154268.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 3 client.154268.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f57257fa700 20 client.154268.objecter dump_active .. 0 homeless 2024-09-18T12:03:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.881+0000 7f573529a700 10 monclient: get_auth_request con 0x7f5714038360 auth_method 0 2024-09-18T12:03:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.885+0000 7f57257fa700 10 client.154268.objecter ms_handle_connect 0x7f5714038360 2024-09-18T12:03:40.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.889+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f572c005560 con 0x7f57300a0750 2024-09-18T12:03:40.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.889+0000 7f57257fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:40.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:40.889+0000 7f57257fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:41.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.049+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f5710000b90 con 0x7f5714038360 2024-09-18T12:03:41.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f57257fa700 1 -- 172.21.15.112:0/1565746112 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (unknown 2057319476 0 0) 0x7f5710000b90 con 0x7f5714038360 2024-09-18T12:03:41.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f573629c700 10 librados: watch_flush enter 2024-09-18T12:03:41.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f573629c700 10 librados: watch_flush exit 2024-09-18T12:03:41.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f573629c700 20 client.154268.objecter shutdown clearing up homeless session... 2024-09-18T12:03:41.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f573629c700 10 client.154268.objecter successfully canceled tick 2024-09-18T12:03:41.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 >> v1:172.21.15.129:6812/20806 conn(0x7f5714038360 legacy=0x7f571403a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f573629c700 10 monclient: shutdown 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f573629c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.053+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 >> v1:172.21.15.112:6790/0 conn(0x7f57300a0750 legacy=0x7f5730136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.057+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 shutdown_connections 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.057+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 >> 172.21.15.112:0/1565746112 conn(0x7f57300a5990 msgr2=0x7f5730138cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.057+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 shutdown_connections 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.057+0000 7f573629c700 1 -- 172.21.15.112:0/1565746112 wait complete. 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.057+0000 7f573629c700 1 librados: shutdown 2024-09-18T12:03:41.060 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T12:03:41.074 INFO:tasks.ceph:Scrubbing osd.1 2024-09-18T12:03:41.074 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:03:41.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.181+0000 7f0d27fff700 10 monclient: get_monmap_and_config 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.181+0000 7f0d27fff700 10 monclient: build_initial_monmap 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.181+0000 7f0d27fff700 10 monclient: monmap: 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:41.187934+0000 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:41.187934+0000 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:41.187 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 1 Processor -- start 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 1 -- start start 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 10 monclient: init 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 10 monclient(hunting): picked mon.noname-c con 0x7f0d2809acc0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 10 monclient(hunting): picked mon.noname-a con 0x7f0d28097530 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 10 monclient(hunting): picked mon.noname-b con 0x7f0d2809cde0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2809b090 con 0x7f0d28097530 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d28063a60 con 0x7f0d2809acc0 2024-09-18T12:03:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d280956d0 con 0x7f0d2809cde0 2024-09-18T12:03:41.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:41.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d27fff700 10 monclient(hunting): authenticate will time out at 1993.028885s 2024-09-18T12:03:41.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d267fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0d28097530 0x7f0d2809c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46092/0 (socket says 172.21.15.112:46092) 2024-09-18T12:03:41.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d267fc700 1 -- 172.21.15.112:0/3749465876 learned_addr learned my addr 172.21.15.112:0/3749465876 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:41.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d267fc700 1 -- 172.21.15.112:0/3749465876 >> v1:172.21.15.112:6789/0 conn(0x7f0d28097530 legacy=0x7f0d2809c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d277fe700 1 -- 172.21.15.112:0/3749465876 >> v1:172.21.15.112:6790/0 conn(0x7f0d2809cde0 legacy=0x7f0d28138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d26ffd700 1 -- 172.21.15.112:0/3749465876 >> v1:172.21.15.129:6789/0 conn(0x7f0d2809acc0 legacy=0x7f0d2809bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208825188 0 0) 0x7f0d2809b090 con 0x7f0d28097530 2024-09-18T12:03:41.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): my global_id is 164172 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d10003570 con 0x7f0d28097530 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441895168 0 0) 0x7f0d280956d0 con 0x7f0d2809cde0 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): my global_id is 154271 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2809b090 con 0x7f0d2809cde0 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313488177 0 0) 0x7f0d28063a60 con 0x7f0d2809acc0 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 10 monclient(hunting): my global_id is 144358 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d280956d0 con 0x7f0d2809acc0 2024-09-18T12:03:41.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.185+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 824361943 0 0) 0x7f0d10003570 con 0x7f0d28097530 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 >> v1:172.21.15.112:6790/0 conn(0x7f0d2809cde0 legacy=0x7f0d28138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 >> v1:172.21.15.129:6789/0 conn(0x7f0d2809acc0 legacy=0x7f0d2809bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 1 monclient: found mon.noname-a 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d28097900 con 0x7f0d28097530 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: _finish_auth 0 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0d1c001940 con 0x7f0d28097530 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: dump: 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.195 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 5 monclient: authenticate success, global_id 164172 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d1c001bd0 con 0x7f0d28097530 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:41.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 1 -- 172.21.15.112:0/3749465876 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0d1c004a90 con 0x7f0d28097530 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 4 set_mon_vals no callback set 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d25ffb700 10 monclient: dump: 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.197 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient: shutdown 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- 172.21.15.112:0/3749465876 >> v1:172.21.15.112:6789/0 conn(0x7f0d28097530 legacy=0x7f0d2809c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- 172.21.15.112:0/3749465876 shutdown_connections 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- 172.21.15.112:0/3749465876 >> 172.21.15.112:0/3749465876 conn(0x7f0d280a15d0 msgr2=0x7f0d280a19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- 172.21.15.112:0/3749465876 shutdown_connections 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- 172.21.15.112:0/3749465876 wait complete. 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient: build_initial_monmap 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient: monmap: 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:41.198 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 librados: starting msgr at 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 librados: starting objecter 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 Processor -- start 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- start start 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 librados: setting wanted keys 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 librados: calling monclient init 2024-09-18T12:03:41.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient: init 2024-09-18T12:03:41.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient(hunting): picked mon.noname-b con 0x7f0d28097530 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient(hunting): picked mon.noname-a con 0x7f0d2809acc0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient(hunting): picked mon.noname-c con 0x7f0d2809cde0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2806fc00 con 0x7f0d2809acc0 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2806ebe0 con 0x7f0d28097530 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d28090170 con 0x7f0d2809cde0 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d27fff700 10 monclient(hunting): authenticate will time out at 1993.035199s 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d267fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0d2809acc0 0x7f0d28146050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46108/0 (socket says 172.21.15.112:46108) 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.189+0000 7f0d267fc700 1 -- 172.21.15.112:0/3829074718 learned_addr learned my addr 172.21.15.112:0/3829074718 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d277fe700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.112:6790/0 conn(0x7f0d2809cde0 legacy=0x7f0d28149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 client.?.objecter ms_handle_connect 0x7f0d2809cde0 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 client.?.objecter ms_handle_connect 0x7f0d2809acc0 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d267fc700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.112:6789/0 conn(0x7f0d2809acc0 legacy=0x7f0d28146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222448472 0 0) 0x7f0d28090170 con 0x7f0d2809cde0 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): my global_id is 154274 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d26ffd700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.129:6789/0 conn(0x7f0d28097530 legacy=0x7f0d280907d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14004f10 con 0x7f0d2809cde0 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 client.?.objecter ms_handle_connect 0x7f0d28097530 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601047127 0 0) 0x7f0d2806fc00 con 0x7f0d2809acc0 2024-09-18T12:03:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): my global_id is 164175 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d28090170 con 0x7f0d2809acc0 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267960137 0 0) 0x7f0d2806ebe0 con 0x7f0d28097530 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): my global_id is 144361 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2806fc00 con 0x7f0d28097530 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343748497 0 0) 0x7f0d28090170 con 0x7f0d2809acc0 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d2806ebe0 con 0x7f0d2809acc0 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0d1c004830 con 0x7f0d2809acc0 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): dump: 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1509014153 0 0) 0x7f0d14004f10 con 0x7f0d2809cde0 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d28090170 con 0x7f0d2809cde0 2024-09-18T12:03:41.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0d20001fc0 con 0x7f0d2809cde0 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): dump: 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d26ffd700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.129:6789/0 conn(0x7f0d28097530 legacy=0x7f0d280907d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d26ffd700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.129:6789/0 conn(0x7f0d28097530 legacy=0x7f0d280907d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d26ffd700 1 --1- 172.21.15.112:0/3829074718 >> v1:172.21.15.129:6789/0 conn(0x7f0d28097530 0x7f0d280907d0 :-1 s=OPENED pgs=9348 cs=1 l=1).handle_message read tag failed 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d26ffd700 1 --1- 172.21.15.112:0/3829074718 >> v1:172.21.15.129:6789/0 conn(0x7f0d28097530 0x7f0d280907d0 :-1 s=OPENED pgs=9348 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.129:6789/0 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1819424798 0 0) 0x7f0d28090170 con 0x7f0d2809cde0 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.129:6789/0 conn(0x7f0d28097530 legacy=0x7f0d280907d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.112:6789/0 conn(0x7f0d2809acc0 legacy=0x7f0d28146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient: _finish_hunting 0 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 monclient: found mon.c 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2808fcf0 con 0x7f0d2809cde0 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient: _finish_auth 0 2024-09-18T12:03:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 5 monclient: authenticate success, global_id 154274 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 10 monclient: _renew_subs 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2809a6b0 con 0x7f0d2809cde0 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 10 client.154274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d20002530 con 0x7f0d2809cde0 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 10 monclient: _renew_subs 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2814aea0 con 0x7f0d2809cde0 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d27fff700 1 librados: init done 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0d20002880 con 0x7f0d2809cde0 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d0effd700 10 monclient: dump: 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.193+0000 7f0d257fa700 4 set_mon_vals no callback set 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d27fff700 10 client.154274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:41.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d27fff700 15 client.154274.objecter _session_command_op_assign -1 1 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d27fff700 20 client.154274.objecter _get_session osd=-1 returning homeless 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d27fff700 10 client.154274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0d20002a00 con 0x7f0d2809cde0 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d280946e0 con 0x7f0d2809cde0 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f0d20043e60 con 0x7f0d2809cde0 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 10 client.154274.objecter ms_dispatch 0x7f0d28000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 3 client.154274.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 3 client.154274.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 10 client.154274.objecter checking command 1 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 20 client.154274.objecter _get_session s=0x7f0d1403ff30 osd=1 3 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 20 client.154274.objecter put_session s=0x7f0d1403ff30 osd=1 3 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 15 client.154274.objecter _session_command_op_remove -1 1 2024-09-18T12:03:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 20 client.154274.objecter _get_session s=0x7f0d1403ff30 osd=1 3 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 20 client.154274.objecter get_session s=0x7f0d1403ff30 osd=1 3 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 15 client.154274.objecter _session_command_op_assign 1 1 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 20 client.154274.objecter put_session s=0x7f0d1403ff30 osd=1 4 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 10 client.154274.objecter _send_command 1 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6800/20985 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d140442e0 con 0x7f0d140405e0 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 20 client.154274.objecter dump_active .. 0 homeless 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_get_version_reply(handle=1 version=699) v2 ==== 24+0+0 (unknown 757610054 0 0) 0x7f0d200442e0 con 0x7f0d2809cde0 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d26ffd700 10 monclient: get_auth_request con 0x7f0d140383a0 auth_method 0 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d267fc700 10 monclient: get_auth_request con 0x7f0d140405e0 auth_method 0 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.197+0000 7f0d0effd700 10 client.154274.objecter ms_handle_connect 0x7f0d140383a0 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.201+0000 7f0d0effd700 10 client.154274.objecter ms_handle_connect 0x7f0d140405e0 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.201+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== osd.1 v1:172.21.15.112:6800/20985 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0d140442e0 con 0x7f0d140405e0 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.201+0000 7f0d0effd700 10 client.154274.objecter ms_dispatch 0x7f0d28000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T12:03:41.215 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.201+0000 7f0d0effd700 10 client.154274.objecter _finish_command 1 = system:0 2024-09-18T12:03:41.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.201+0000 7f0d0effd700 20 client.154274.objecter put_session s=0x7f0d1403ff30 osd=1 4 2024-09-18T12:03:41.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.201+0000 7f0d0effd700 15 client.154274.objecter _session_command_op_remove 1 1 2024-09-18T12:03:41.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 10 client.154274.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-18T12:03:41.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 15 client.154274.objecter _session_command_op_assign -1 2 2024-09-18T12:03:41.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 20 client.154274.objecter _get_session s=0x7f0d1403ff30 osd=1 3 2024-09-18T12:03:41.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 20 client.154274.objecter put_session s=0x7f0d1403ff30 osd=1 3 2024-09-18T12:03:41.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 20 client.154274.objecter _get_session s=0x7f0d1403ff30 osd=1 3 2024-09-18T12:03:41.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 15 client.154274.objecter _session_command_op_remove -1 2 2024-09-18T12:03:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 20 client.154274.objecter get_session s=0x7f0d1403ff30 osd=1 3 2024-09-18T12:03:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 15 client.154274.objecter _session_command_op_assign 1 2 2024-09-18T12:03:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 20 client.154274.objecter put_session s=0x7f0d1403ff30 osd=1 4 2024-09-18T12:03:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 10 client.154274.objecter _send_command 2 2024-09-18T12:03:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 --> v1:172.21.15.112:6800/20985 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0d2805f2e0 con 0x7f0d140405e0 2024-09-18T12:03:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d0effd700 1 -- 172.21.15.112:0/3829074718 <== osd.1 v1:172.21.15.112:6800/20985 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f0d2805f2e0 con 0x7f0d140405e0 2024-09-18T12:03:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d0effd700 10 client.154274.objecter ms_dispatch 0x7f0d28000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T12:03:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d0effd700 10 client.154274.objecter _finish_command 2 = system:0 2024-09-18T12:03:41.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d0effd700 20 client.154274.objecter put_session s=0x7f0d1403ff30 osd=1 4 2024-09-18T12:03:41.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d0effd700 15 client.154274.objecter _session_command_op_remove 1 2 2024-09-18T12:03:41.226 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:41.226 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:03:41.226 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:41.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 10 librados: watch_flush enter 2024-09-18T12:03:41.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 10 librados: watch_flush exit 2024-09-18T12:03:41.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 10 client.154274.objecter close_session for osd.1 2024-09-18T12:03:41.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.112:6800/20985 conn(0x7f0d140405e0 legacy=0x7f0d14042a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 20 client.154274.objecter put_session s=0x7f0d1403ff30 osd=1 1 2024-09-18T12:03:41.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 20 client.154274.objecter shutdown clearing up homeless session... 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 10 client.154274.objecter successfully canceled tick 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.129:6812/20806 conn(0x7f0d140383a0 legacy=0x7f0d1403a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 10 monclient: shutdown 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 >> v1:172.21.15.112:6790/0 conn(0x7f0d2809cde0 legacy=0x7f0d28149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d277fe700 1 -- 172.21.15.112:0/3829074718 reap_dead start 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 shutdown_connections 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 >> 172.21.15.112:0/3829074718 conn(0x7f0d280a15d0 msgr2=0x7f0d2806c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 shutdown_connections 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 -- 172.21.15.112:0/3829074718 wait complete. 2024-09-18T12:03:41.228 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.221+0000 7f0d27fff700 1 librados: shutdown 2024-09-18T12:03:41.241 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.345+0000 7fcc178af700 10 monclient: get_monmap_and_config 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.345+0000 7fcc178af700 10 monclient: build_initial_monmap 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.345+0000 7fcc178af700 10 monclient: monmap: 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:41.352113+0000 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:41.352113+0000 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:41.351 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:41.352 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:41.352 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 1 Processor -- start 2024-09-18T12:03:41.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 1 -- start start 2024-09-18T12:03:41.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 10 monclient: init 2024-09-18T12:03:41.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:41.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 10 monclient(hunting): picked mon.noname-a con 0x7fcc10130420 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 10 monclient(hunting): picked mon.noname-c con 0x7fcc1006ea60 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 10 monclient(hunting): picked mon.noname-b con 0x7fcc10136aa0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc10130f00 con 0x7fcc10130420 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1006c990 con 0x7fcc1006ea60 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100a12c0 con 0x7fcc10136aa0 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc178af700 10 monclient(hunting): authenticate will time out at 1993.193753s 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc1564b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcc10130420 0x7fcc101307f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46114/0 (socket says 172.21.15.112:46114) 2024-09-18T12:03:41.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc1564b700 1 -- 172.21.15.112:0/2847629668 learned_addr learned my addr 172.21.15.112:0/2847629668 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:41.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.349+0000 7fcc1564b700 1 -- 172.21.15.112:0/2847629668 >> v1:172.21.15.112:6789/0 conn(0x7fcc10130420 legacy=0x7fcc101307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc15e4c700 1 -- 172.21.15.112:0/2847629668 >> v1:172.21.15.112:6790/0 conn(0x7fcc10136aa0 legacy=0x7fcc10138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc14e4a700 1 -- 172.21.15.112:0/2847629668 >> v1:172.21.15.129:6789/0 conn(0x7fcc1006ea60 legacy=0x7fcc10135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018778252 0 0) 0x7fcc10130f00 con 0x7fcc10130420 2024-09-18T12:03:41.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): my global_id is 164178 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbfc003570 con 0x7fcc10130420 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740359220 0 0) 0x7fcc100a12c0 con 0x7fcc10136aa0 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): my global_id is 154277 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc10130f00 con 0x7fcc10136aa0 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458930070 0 0) 0x7fcc1006c990 con 0x7fcc1006ea60 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient(hunting): my global_id is 144364 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc100a12c0 con 0x7fcc1006ea60 2024-09-18T12:03:41.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 490556012 0 0) 0x7fcc10130f00 con 0x7fcc10136aa0 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 >> v1:172.21.15.129:6789/0 conn(0x7fcc1006ea60 legacy=0x7fcc10135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 >> v1:172.21.15.112:6789/0 conn(0x7fcc10130420 legacy=0x7fcc101307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient: _finish_hunting 0 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 monclient: found mon.noname-b 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1006d7d0 con 0x7fcc10136aa0 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient: _finish_auth 0 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 5 monclient: authenticate success, global_id 154277 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcc08001cf0 con 0x7fcc10136aa0 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient: dump: 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc15e4c700 1 --1- 172.21.15.112:0/2847629668 >> v1:172.21.15.112:6790/0 conn(0x7fcc10136aa0 0x7fcc10138e80 :-1 s=OPENED pgs=9361 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7fcc08001f80 config(0 keys) v1 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc08001f80 con 0x7fcc10136aa0 2024-09-18T12:03:41.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:41.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:41.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 4 set_mon_vals no callback set 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient: shutdown 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- 172.21.15.112:0/2847629668 >> v1:172.21.15.112:6790/0 conn(0x7fcc10136aa0 legacy=0x7fcc10138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 1 -- 172.21.15.112:0/2847629668 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcc08002420 con 0x7fcc10136aa0 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc03fff700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- 172.21.15.112:0/2847629668 shutdown_connections 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- 172.21.15.112:0/2847629668 >> 172.21.15.112:0/2847629668 conn(0x7fcc100680d0 msgr2=0x7fcc100a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- 172.21.15.112:0/2847629668 shutdown_connections 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- 172.21.15.112:0/2847629668 wait complete. 2024-09-18T12:03:41.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient: build_initial_monmap 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient: monmap: 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 librados: starting msgr at 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 librados: starting objecter 2024-09-18T12:03:41.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 Processor -- start 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- start start 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 librados: setting wanted keys 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 librados: calling monclient init 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient: init 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient(hunting): picked mon.noname-b con 0x7fcc10130420 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient(hunting): picked mon.noname-c con 0x7fcc10136aa0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient(hunting): picked mon.noname-a con 0x7fcc1006c250 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1009bce0 con 0x7fcc1006c250 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1009dd50 con 0x7fcc10130420 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100994a0 con 0x7fcc10136aa0 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc178af700 10 monclient(hunting): authenticate will time out at 1993.199744s 2024-09-18T12:03:41.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc14e4a700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fcc10136aa0 0x7fcc1006ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58770/0 (socket says 172.21.15.112:58770) 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.353+0000 7fcc14e4a700 1 -- 172.21.15.112:0/1533328309 learned_addr learned my addr 172.21.15.112:0/1533328309 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc14e4a700 1 -- 172.21.15.112:0/1533328309 >> v1:172.21.15.112:6790/0 conn(0x7fcc10136aa0 legacy=0x7fcc1006ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 client.?.objecter ms_handle_connect 0x7fcc10136aa0 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 client.?.objecter ms_handle_connect 0x7fcc1006c250 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc15e4c700 1 -- 172.21.15.112:0/1533328309 >> v1:172.21.15.112:6789/0 conn(0x7fcc1006c250 legacy=0x7fcc10149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972608909 0 0) 0x7fcc100994a0 con 0x7fcc10136aa0 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): my global_id is 154280 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf4004f10 con 0x7fcc10136aa0 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 client.?.objecter ms_handle_connect 0x7fcc10130420 2024-09-18T12:03:41.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc1564b700 1 -- 172.21.15.112:0/1533328309 >> v1:172.21.15.129:6789/0 conn(0x7fcc10130420 legacy=0x7fcc1006a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873985523 0 0) 0x7fcc1009bce0 con 0x7fcc1006c250 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): my global_id is 164181 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc100994a0 con 0x7fcc1006c250 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499392075 0 0) 0x7fcc1009dd50 con 0x7fcc10130420 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): my global_id is 144367 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1009bce0 con 0x7fcc10130420 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3115749302 0 0) 0x7fcbf4004f10 con 0x7fcc10136aa0 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc1009dd50 con 0x7fcc10136aa0 2024-09-18T12:03:41.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcc04001c30 con 0x7fcc10136aa0 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): dump: 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 836958754 0 0) 0x7fcc100994a0 con 0x7fcc1006c250 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbf4004f10 con 0x7fcc1006c250 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcc08001c70 con 0x7fcc1006c250 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): dump: 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.371 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819420555 0 0) 0x7fcc1009bce0 con 0x7fcc10130420 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc100994a0 con 0x7fcc10130420 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcc0c001360 con 0x7fcc10130420 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient(hunting): dump: 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.372 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2125293785 0 0) 0x7fcc1009dd50 con 0x7fcc10136aa0 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 >> v1:172.21.15.129:6789/0 conn(0x7fcc10130420 legacy=0x7fcc1006a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 >> v1:172.21.15.112:6789/0 conn(0x7fcc1006c250 legacy=0x7fcc10149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient: _finish_hunting 0 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 monclient: found mon.c 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1006b5a0 con 0x7fcc10136aa0 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient: _finish_auth 0 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:41.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 5 monclient: authenticate success, global_id 154280 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 10 monclient: _renew_subs 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1006bb90 con 0x7fcc10136aa0 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 10 client.154280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 10 monclient: _renew_subs 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc10135570 con 0x7fcc10136aa0 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc178af700 1 librados: init done 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc040021b0 con 0x7fcc10136aa0 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fcc04002580 con 0x7fcc10136aa0 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc017fa700 10 monclient: dump: 2024-09-18T12:03:41.374 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.357+0000 7fcc02ffd700 4 set_mon_vals no callback set 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fcc040026e0 con 0x7fcc10136aa0 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fcc04043bf0 con 0x7fcc10136aa0 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc017fa700 10 client.154280.objecter ms_dispatch 0x7fcc10000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc017fa700 3 client.154280.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:41.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc017fa700 3 client.154280.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:41.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc017fa700 20 client.154280.objecter dump_active .. 0 homeless 2024-09-18T12:03:41.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc178af700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:41.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc178af700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:41.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc178af700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf00050b0 con 0x7fcc10136aa0 2024-09-18T12:03:41.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.361+0000 7fcc1564b700 10 monclient: get_auth_request con 0x7fcbf4038600 auth_method 0 2024-09-18T12:03:41.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.365+0000 7fcc017fa700 10 client.154280.objecter ms_handle_connect 0x7fcbf4038600 2024-09-18T12:03:41.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.369+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fcc04014a50 con 0x7fcc10136aa0 2024-09-18T12:03:41.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.369+0000 7fcc017fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:41.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.369+0000 7fcc017fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:41.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.541+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbf0000b90 con 0x7fcbf4038600 2024-09-18T12:03:41.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.541+0000 7fcc017fa700 1 -- 172.21.15.112:0/1533328309 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (unknown 489605429 0 0) 0x7fcbf0000b90 con 0x7fcbf4038600 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 10 librados: watch_flush enter 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 10 librados: watch_flush exit 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 20 client.154280.objecter shutdown clearing up homeless session... 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 10 client.154280.objecter successfully canceled tick 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 >> v1:172.21.15.129:6812/20806 conn(0x7fcbf4038600 legacy=0x7fcbf403aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 10 monclient: shutdown 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 >> v1:172.21.15.112:6790/0 conn(0x7fcc10136aa0 legacy=0x7fcc1006ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 shutdown_connections 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 >> 172.21.15.112:0/1533328309 conn(0x7fcc100680d0 msgr2=0x7fcc1012c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 shutdown_connections 2024-09-18T12:03:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 1 -- 172.21.15.112:0/1533328309 wait complete. 2024-09-18T12:03:41.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.545+0000 7fcc178af700 1 librados: shutdown 2024-09-18T12:03:41.550 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T12:03:41.563 INFO:tasks.ceph:Scrubbing osd.2 2024-09-18T12:03:41.563 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient: get_monmap_and_config 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient: build_initial_monmap 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient: monmap: 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:41.678110+0000 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:41.678110+0000 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.677 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:41.678 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:41.678 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:41.678 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 1 Processor -- start 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 1 -- start start 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient: init 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient(hunting): picked mon.noname-c con 0x7fe4b00a21c0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient(hunting): picked mon.noname-a con 0x7fe4b01310c0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient(hunting): picked mon.noname-b con 0x7fe4b0134bc0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:41.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b00a2590 con 0x7fe4b01310c0 2024-09-18T12:03:41.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b006b090 con 0x7fe4b00a21c0 2024-09-18T12:03:41.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b006a7c0 con 0x7fe4b0134bc0 2024-09-18T12:03:41.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:41.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4b58fa700 10 monclient(hunting): authenticate will time out at 1993.519487s 2024-09-18T12:03:41.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4ae7fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe4b01310c0 0x7fe4b01334a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46134/0 (socket says 172.21.15.112:46134) 2024-09-18T12:03:41.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.673+0000 7fe4ae7fc700 1 -- 172.21.15.112:0/2316781692 learned_addr learned my addr 172.21.15.112:0/2316781692 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4ae7fc700 1 -- 172.21.15.112:0/2316781692 >> v1:172.21.15.112:6789/0 conn(0x7fe4b01310c0 legacy=0x7fe4b01334a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4af7fe700 1 -- 172.21.15.112:0/2316781692 >> v1:172.21.15.112:6790/0 conn(0x7fe4b0134bc0 legacy=0x7fe4b0136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4aeffd700 1 -- 172.21.15.112:0/2316781692 >> v1:172.21.15.129:6789/0 conn(0x7fe4b00a21c0 legacy=0x7fe4b006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630418268 0 0) 0x7fe4b00a2590 con 0x7fe4b01310c0 2024-09-18T12:03:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient(hunting): my global_id is 164184 2024-09-18T12:03:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe498003570 con 0x7fe4b01310c0 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 613433205 0 0) 0x7fe498003570 con 0x7fe4b01310c0 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 >> v1:172.21.15.112:6790/0 conn(0x7fe4b0134bc0 legacy=0x7fe4b0136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 >> v1:172.21.15.129:6789/0 conn(0x7fe4b00a21c0 legacy=0x7fe4b006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient: _finish_hunting 0 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 1 monclient: found mon.noname-a 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b006bde0 con 0x7fe4b01310c0 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient: _finish_auth 0 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe4a0001940 con 0x7fe4b01310c0 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4adffb700 10 monclient: dump: 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4b58fa700 5 monclient: authenticate success, global_id 164184 2024-09-18T12:03:41.684 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.677+0000 7fe4b58fa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4a0001c10 con 0x7fe4b01310c0 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4adffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4adffb700 1 -- 172.21.15.112:0/2316781692 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe4a0004a90 con 0x7fe4b01310c0 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4adffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4adffb700 10 monclient: dump: 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.687 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.687 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.687 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.687 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.687 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.687 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:41.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 4 set_mon_vals no callback set 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 10 monclient: shutdown 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 1 -- 172.21.15.112:0/2316781692 >> v1:172.21.15.112:6789/0 conn(0x7fe4b01310c0 legacy=0x7fe4b01334a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 1 -- 172.21.15.112:0/2316781692 shutdown_connections 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 1 -- 172.21.15.112:0/2316781692 >> 172.21.15.112:0/2316781692 conn(0x7fe4b00680d0 msgr2=0x7fe4b00a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 1 -- 172.21.15.112:0/2316781692 shutdown_connections 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.681+0000 7fe4b58fa700 1 -- 172.21.15.112:0/2316781692 wait complete. 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient: build_initial_monmap 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient: monmap: 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:41.689 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 librados: starting msgr at 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 librados: starting objecter 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 Processor -- start 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 -- start start 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 librados: setting wanted keys 2024-09-18T12:03:41.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 librados: calling monclient init 2024-09-18T12:03:41.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient: init 2024-09-18T12:03:41.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient(hunting): picked mon.noname-c con 0x7fe4b00a21c0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient(hunting): picked mon.noname-b con 0x7fe4b01310c0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient(hunting): picked mon.noname-a con 0x7fe4b0134bc0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b009ef90 con 0x7fe4b0134bc0 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b009a680 con 0x7fe4b01310c0 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b01322b0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4aeffd700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe4b00a21c0 0x7fe4b0131490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58780/0 (socket says 172.21.15.112:58780) 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4aeffd700 1 -- 172.21.15.112:0/590965568 learned_addr learned my addr 172.21.15.112:0/590965568 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.685+0000 7fe4b58fa700 10 monclient(hunting): authenticate will time out at 1993.530962s 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4aeffd700 1 -- 172.21.15.112:0/590965568 >> v1:172.21.15.112:6790/0 conn(0x7fe4b00a21c0 legacy=0x7fe4b0131490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 client.?.objecter ms_handle_connect 0x7fe4b00a21c0 2024-09-18T12:03:41.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4af7fe700 1 -- 172.21.15.112:0/590965568 >> v1:172.21.15.112:6789/0 conn(0x7fe4b0134bc0 legacy=0x7fe4b014a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 client.?.objecter ms_handle_connect 0x7fe4b0134bc0 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 client.?.objecter ms_handle_connect 0x7fe4b01310c0 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4ae7fc700 1 -- 172.21.15.112:0/590965568 >> v1:172.21.15.129:6789/0 conn(0x7fe4b01310c0 legacy=0x7fe4b0131ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936757878 0 0) 0x7fe4b01322b0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): my global_id is 154283 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c004f10 con 0x7fe4b00a21c0 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67632070 0 0) 0x7fe4b009ef90 con 0x7fe4b0134bc0 2024-09-18T12:03:41.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): my global_id is 164187 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b01322b0 con 0x7fe4b0134bc0 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4ae7fc700 1 --1- 172.21.15.112:0/590965568 >> v1:172.21.15.129:6789/0 conn(0x7fe4b01310c0 0x7fe4b0131ba0 :-1 s=OPENED pgs=9352 cs=1 l=1).queue_received will delay after 1.72666e+09 on 0x7fe4b009a680 auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72011786 0 0) 0x7fe49c004f10 con 0x7fe4b00a21c0 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b009ef90 con 0x7fe4b00a21c0 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe4a4001d20 con 0x7fe4b00a21c0 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): dump: 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.697 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4075429380 0 0) 0x7fe4b01322b0 con 0x7fe4b0134bc0 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe49c004f10 con 0x7fe4b0134bc0 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe4a8001eb0 con 0x7fe4b0134bc0 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient(hunting): dump: 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1488097866 0 0) 0x7fe4b009ef90 con 0x7fe4b00a21c0 2024-09-18T12:03:41.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 >> v1:172.21.15.129:6789/0 conn(0x7fe4b01310c0 legacy=0x7fe4b0131ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 >> v1:172.21.15.112:6789/0 conn(0x7fe4b0134bc0 legacy=0x7fe4b014a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient: _finish_hunting 0 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 monclient: found mon.c 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b00994b0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient: _finish_auth 0 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860050628 0 0) 0x7fe4b009a680 con 0x7fe4b01310c0 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 5 monclient: authenticate success, global_id 154283 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 10 monclient: _renew_subs 2024-09-18T12:03:41.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b009a3a0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 10 client.154283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 10 monclient: _renew_subs 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b009b060 con 0x7fe4b00a21c0 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4b58fa700 1 librados: init done 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4a40022d0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe4a40025e0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe4ad7fa700 4 set_mon_vals no callback set 2024-09-18T12:03:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 10 monclient: dump: 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.689+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe4a40027f0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe4b58fa700 10 client.154283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:41.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe4b58fa700 15 client.154283.objecter _session_command_op_assign -1 1 2024-09-18T12:03:41.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe4b58fa700 20 client.154283.objecter _get_session osd=-1 returning homeless 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe4b58fa700 10 client.154283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe4b58fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe4900028c0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fe4a4043c00 con 0x7fe4b00a21c0 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 10 client.154283.objecter ms_dispatch 0x7fe4b0000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 3 client.154283.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 3 client.154283.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 10 client.154283.objecter checking command 1 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 20 client.154283.objecter _get_session s=0x7fe49c03fdc0 osd=2 3 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 20 client.154283.objecter put_session s=0x7fe49c03fdc0 osd=2 3 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 15 client.154283.objecter _session_command_op_remove -1 1 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 20 client.154283.objecter _get_session s=0x7fe49c03fdc0 osd=2 3 2024-09-18T12:03:41.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 20 client.154283.objecter get_session s=0x7fe49c03fdc0 osd=2 3 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 15 client.154283.objecter _session_command_op_assign 2 1 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 20 client.154283.objecter put_session s=0x7fe49c03fdc0 osd=2 4 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 10 client.154283.objecter _send_command 1 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6808/20986 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe49c044170 con 0x7fe49c040470 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 20 client.154283.objecter dump_active .. 0 homeless 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_get_version_reply(handle=1 version=699) v2 ==== 24+0+0 (unknown 757610054 0 0) 0x7fe4a4014ac0 con 0x7fe4b00a21c0 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe4ae7fc700 10 monclient: get_auth_request con 0x7fe49c038320 auth_method 0 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe4af7fe700 10 monclient: get_auth_request con 0x7fe49c040470 auth_method 0 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.693+0000 7fe496ffd700 10 client.154283.objecter ms_handle_connect 0x7fe49c038320 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.697+0000 7fe496ffd700 10 client.154283.objecter ms_handle_connect 0x7fe49c040470 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.697+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== osd.2 v1:172.21.15.112:6808/20986 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe49c044170 con 0x7fe49c040470 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.697+0000 7fe496ffd700 10 client.154283.objecter ms_dispatch 0x7fe4b0000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.697+0000 7fe496ffd700 10 client.154283.objecter _finish_command 1 = system:0 2024-09-18T12:03:41.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.697+0000 7fe496ffd700 20 client.154283.objecter put_session s=0x7fe49c03fdc0 osd=2 4 2024-09-18T12:03:41.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.697+0000 7fe496ffd700 15 client.154283.objecter _session_command_op_remove 2 1 2024-09-18T12:03:41.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 10 client.154283.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 15 client.154283.objecter _session_command_op_assign -1 2 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 20 client.154283.objecter _get_session s=0x7fe49c03fdc0 osd=2 3 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 20 client.154283.objecter put_session s=0x7fe49c03fdc0 osd=2 3 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 20 client.154283.objecter _get_session s=0x7fe49c03fdc0 osd=2 3 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 15 client.154283.objecter _session_command_op_remove -1 2 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 20 client.154283.objecter get_session s=0x7fe49c03fdc0 osd=2 3 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 15 client.154283.objecter _session_command_op_assign 2 2 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 20 client.154283.objecter put_session s=0x7fe49c03fdc0 osd=2 4 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 10 client.154283.objecter _send_command 2 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 --> v1:172.21.15.112:6808/20986 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe490001210 con 0x7fe49c040470 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe496ffd700 1 -- 172.21.15.112:0/590965568 <== osd.2 v1:172.21.15.112:6808/20986 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7fe490001210 con 0x7fe49c040470 2024-09-18T12:03:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe496ffd700 10 client.154283.objecter ms_dispatch 0x7fe4b0000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T12:03:41.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe496ffd700 10 client.154283.objecter _finish_command 2 = system:0 2024-09-18T12:03:41.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe496ffd700 20 client.154283.objecter put_session s=0x7fe49c03fdc0 osd=2 4 2024-09-18T12:03:41.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe496ffd700 15 client.154283.objecter _session_command_op_remove 2 2 2024-09-18T12:03:41.721 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:41.721 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:03:41.721 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 10 librados: watch_flush enter 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 10 librados: watch_flush exit 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 10 client.154283.objecter close_session for osd.2 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 >> v1:172.21.15.112:6808/20986 conn(0x7fe49c040470 legacy=0x7fe49c042900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 20 client.154283.objecter put_session s=0x7fe49c03fdc0 osd=2 1 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 20 client.154283.objecter shutdown clearing up homeless session... 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 10 client.154283.objecter successfully canceled tick 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 >> v1:172.21.15.129:6812/20806 conn(0x7fe49c038320 legacy=0x7fe49c03a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 10 monclient: shutdown 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 >> v1:172.21.15.112:6790/0 conn(0x7fe4b00a21c0 legacy=0x7fe4b0131490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4af7fe700 1 -- 172.21.15.112:0/590965568 reap_dead start 2024-09-18T12:03:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 shutdown_connections 2024-09-18T12:03:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 >> 172.21.15.112:0/590965568 conn(0x7fe4b00680d0 msgr2=0x7fe4b0130590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 shutdown_connections 2024-09-18T12:03:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 -- 172.21.15.112:0/590965568 wait complete. 2024-09-18T12:03:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.717+0000 7fe4b58fa700 1 librados: shutdown 2024-09-18T12:03:41.732 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.841+0000 7fa4ddc26700 10 monclient: get_monmap_and_config 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.841+0000 7fa4ddc26700 10 monclient: build_initial_monmap 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.841+0000 7fa4ddc26700 10 monclient: monmap: 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:41.847175+0000 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:41.847175+0000 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:41.846 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 1 Processor -- start 2024-09-18T12:03:41.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 1 -- start start 2024-09-18T12:03:41.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 10 monclient: init 2024-09-18T12:03:41.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:41.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:41.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 10 monclient(hunting): picked mon.noname-c con 0x7fa4d8097020 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:41.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 10 monclient(hunting): picked mon.noname-b con 0x7fa4d80973f0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:41.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 10 monclient(hunting): picked mon.noname-a con 0x7fa4d8135740 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:41.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d809c2a0 con 0x7fa4d8135740 2024-09-18T12:03:41.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d8098050 con 0x7fa4d8097020 2024-09-18T12:03:41.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d809e5a0 con 0x7fa4d80973f0 2024-09-18T12:03:41.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:41.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4ddc26700 10 monclient(hunting): authenticate will time out at 1993.688749s 2024-09-18T12:03:41.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d7fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fa4d80973f0 0x7fa4d806f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58794/0 (socket says 172.21.15.112:58794) 2024-09-18T12:03:41.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d7fff700 1 -- 172.21.15.112:0/974860307 learned_addr learned my addr 172.21.15.112:0/974860307 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:41.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d7fff700 1 -- 172.21.15.112:0/974860307 >> v1:172.21.15.112:6790/0 conn(0x7fa4d80973f0 legacy=0x7fa4d806f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4dd425700 1 -- 172.21.15.112:0/974860307 >> v1:172.21.15.112:6789/0 conn(0x7fa4d8135740 legacy=0x7fa4d8137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506738551 0 0) 0x7fa4d809e5a0 con 0x7fa4d80973f0 2024-09-18T12:03:41.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): my global_id is 154286 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c0003570 con 0x7fa4d80973f0 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4dcc24700 1 -- 172.21.15.112:0/974860307 >> v1:172.21.15.129:6789/0 conn(0x7fa4d8097020 legacy=0x7fa4d8067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43831056 0 0) 0x7fa4d809c2a0 con 0x7fa4d8135740 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): my global_id is 164190 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d809e5a0 con 0x7fa4d8135740 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189062563 0 0) 0x7fa4d8098050 con 0x7fa4d8097020 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 10 monclient(hunting): my global_id is 144373 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d809c2a0 con 0x7fa4d8097020 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1471154894 0 0) 0x7fa4c0003570 con 0x7fa4d80973f0 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 >> v1:172.21.15.129:6789/0 conn(0x7fa4d8097020 legacy=0x7fa4d8067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.845+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 >> v1:172.21.15.112:6789/0 conn(0x7fa4d8135740 legacy=0x7fa4d8137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: _finish_hunting 0 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 1 monclient: found mon.noname-b 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d8097d70 con 0x7fa4d80973f0 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: _finish_auth 0 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 5 monclient: authenticate success, global_id 154286 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa4cc001a00 con 0x7fa4d80973f0 2024-09-18T12:03:41.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:41.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: dump: 2024-09-18T12:03:41.855 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.855 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.855 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.855 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.855 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4cc001b60 con 0x7fa4d80973f0 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 4 set_mon_vals no callback set 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 1 -- 172.21.15.112:0/974860307 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa4cc0048f0 con 0x7fa4d80973f0 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4d77fe700 10 monclient: dump: 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 10 monclient: shutdown 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:41.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 -- 172.21.15.112:0/974860307 >> v1:172.21.15.112:6790/0 conn(0x7fa4d80973f0 legacy=0x7fa4d806f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 -- 172.21.15.112:0/974860307 shutdown_connections 2024-09-18T12:03:41.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 -- 172.21.15.112:0/974860307 >> 172.21.15.112:0/974860307 conn(0x7fa4d80a1e30 msgr2=0x7fa4d80a2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:41.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 -- 172.21.15.112:0/974860307 shutdown_connections 2024-09-18T12:03:41.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 -- 172.21.15.112:0/974860307 wait complete. 2024-09-18T12:03:41.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 10 monclient: build_initial_monmap 2024-09-18T12:03:41.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 10 monclient: monmap: 2024-09-18T12:03:41.863 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:41.863 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:41.863 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:41.863 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:41.863 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:41.863 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 librados: starting msgr at 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 librados: starting objecter 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 Processor -- start 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 -- start start 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 librados: setting wanted keys 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 1 librados: calling monclient init 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.849+0000 7fa4ddc26700 10 monclient: init 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:41.864 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 10 monclient(hunting): picked mon.noname-c con 0x7fa4d8097020 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 10 monclient(hunting): picked mon.noname-a con 0x7fa4d80973f0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 10 monclient(hunting): picked mon.noname-b con 0x7fa4d8135740 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d809a6f0 con 0x7fa4d80973f0 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d806dc60 con 0x7fa4d8135740 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d81421a0 con 0x7fa4d8097020 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4ddc26700 10 monclient(hunting): authenticate will time out at 1993.696742s 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d7fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa4d80973f0 0x7fa4d8141a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46164/0 (socket says 172.21.15.112:46164) 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d7fff700 1 -- 172.21.15.112:0/2263931946 learned_addr learned my addr 172.21.15.112:0/2263931946 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4dcc24700 1 -- 172.21.15.112:0/2263931946 >> v1:172.21.15.112:6790/0 conn(0x7fa4d8097020 legacy=0x7fa4d8141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d7fff700 1 -- 172.21.15.112:0/2263931946 >> v1:172.21.15.112:6789/0 conn(0x7fa4d80973f0 legacy=0x7fa4d8141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 client.?.objecter ms_handle_connect 0x7fa4d8097020 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 client.?.objecter ms_handle_connect 0x7fa4d80973f0 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845410872 0 0) 0x7fa4d81421a0 con 0x7fa4d8097020 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): my global_id is 154289 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4dd425700 1 -- 172.21.15.112:0/2263931946 >> v1:172.21.15.129:6789/0 conn(0x7fa4d8135740 legacy=0x7fa4d81499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c4004f10 con 0x7fa4d8097020 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 client.?.objecter ms_handle_connect 0x7fa4d8135740 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652973125 0 0) 0x7fa4d809a6f0 con 0x7fa4d80973f0 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): my global_id is 164193 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d81421a0 con 0x7fa4d80973f0 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700361481 0 0) 0x7fa4d806dc60 con 0x7fa4d8135740 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 10 monclient(hunting): my global_id is 144376 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d809a6f0 con 0x7fa4d8135740 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.853+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3861003424 0 0) 0x7fa4d81421a0 con 0x7fa4d80973f0 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4d806dc60 con 0x7fa4d80973f0 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa4cc001360 con 0x7fa4d80973f0 2024-09-18T12:03:41.867 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): dump: 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1372296028 0 0) 0x7fa4c4004f10 con 0x7fa4d8097020 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4d81421a0 con 0x7fa4d8097020 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa4c8001c40 con 0x7fa4d8097020 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): dump: 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1218546700 0 0) 0x7fa4d809a6f0 con 0x7fa4d8135740 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4c4004f10 con 0x7fa4d8135740 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa4d0001d40 con 0x7fa4d8135740 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient(hunting): dump: 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.870 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1806293618 0 0) 0x7fa4d806dc60 con 0x7fa4d80973f0 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 >> v1:172.21.15.112:6790/0 conn(0x7fa4d8097020 legacy=0x7fa4d8141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 >> v1:172.21.15.129:6789/0 conn(0x7fa4d8135740 legacy=0x7fa4d81499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient: _finish_hunting 0 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 monclient: found mon.a 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:41.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d806d980 con 0x7fa4d80973f0 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient: _finish_auth 0 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 5 monclient: authenticate success, global_id 164193 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 10 monclient: _renew_subs 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d806c7b0 con 0x7fa4d80973f0 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 10 client.164193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 10 monclient: _renew_subs 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d8134c70 con 0x7fa4d80973f0 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 1 librados: init done 2024-09-18T12:03:41.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4cc004930 con 0x7fa4d80973f0 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fa4cc0050f0 con 0x7fa4d80973f0 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 monclient: dump: 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:41.873 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d6ffd700 4 set_mon_vals no callback set 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b80050b0 con 0x7fa4d80973f0 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fa4cc00f970 con 0x7fa4d80973f0 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fa4cc043b70 con 0x7fa4d80973f0 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 10 client.164193.objecter ms_dispatch 0x7fa4d8000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 3 client.164193.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 3 client.164193.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4d4ff9700 20 client.164193.objecter dump_active .. 0 homeless 2024-09-18T12:03:41.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.857+0000 7fa4dcc24700 10 monclient: get_auth_request con 0x7fa4c4038730 auth_method 0 2024-09-18T12:03:41.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.861+0000 7fa4d4ff9700 10 client.164193.objecter ms_handle_connect 0x7fa4c4038730 2024-09-18T12:03:41.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.865+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fa4cc0146d0 con 0x7fa4d80973f0 2024-09-18T12:03:41.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.869+0000 7fa4d4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:41.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:41.869+0000 7fa4d4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:42.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.029+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4b8000b90 con 0x7fa4c4038730 2024-09-18T12:03:42.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4d4ff9700 1 -- 172.21.15.112:0/2263931946 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (unknown 3049115190 0 0) 0x7fa4b8000b90 con 0x7fa4c4038730 2024-09-18T12:03:42.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 10 librados: watch_flush enter 2024-09-18T12:03:42.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 10 librados: watch_flush exit 2024-09-18T12:03:42.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 20 client.164193.objecter shutdown clearing up homeless session... 2024-09-18T12:03:42.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 10 client.164193.objecter successfully canceled tick 2024-09-18T12:03:42.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 >> v1:172.21.15.129:6812/20806 conn(0x7fa4c4038730 legacy=0x7fa4c403abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 10 monclient: shutdown 2024-09-18T12:03:42.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:42.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 >> v1:172.21.15.112:6789/0 conn(0x7fa4d80973f0 legacy=0x7fa4d8141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 shutdown_connections 2024-09-18T12:03:42.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 >> 172.21.15.112:0/2263931946 conn(0x7fa4d80a1e30 msgr2=0x7fa4d80a0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:42.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 shutdown_connections 2024-09-18T12:03:42.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.033+0000 7fa4ddc26700 1 -- 172.21.15.112:0/2263931946 wait complete. 2024-09-18T12:03:42.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.037+0000 7fa4ddc26700 1 librados: shutdown 2024-09-18T12:03:42.040 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T12:03:42.053 INFO:tasks.ceph:Scrubbing osd.3 2024-09-18T12:03:42.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.161+0000 7f646ee3c700 10 monclient: get_monmap_and_config 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.161+0000 7f646ee3c700 10 monclient: build_initial_monmap 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.161+0000 7f646ee3c700 10 monclient: monmap: 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:42.167187+0000 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:42.167187+0000 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:42.166 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.167 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:42.167 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:42.167 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:42.167 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 1 Processor -- start 2024-09-18T12:03:42.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 1 -- start start 2024-09-18T12:03:42.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 10 monclient: init 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 10 monclient(hunting): picked mon.noname-a con 0x7f64680a0380 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 10 monclient(hunting): picked mon.noname-c con 0x7f64680a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 10 monclient(hunting): picked mon.noname-b con 0x7f6468097b70 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6468097f90 con 0x7f64680a0380 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64680a1550 con 0x7f64680a0e60 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6468096dd0 con 0x7f6468097b70 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ee3c700 10 monclient(hunting): authenticate will time out at 1994.009218s 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646de3a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f64680a0380 0x7f64680a0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46176/0 (socket says 172.21.15.112:46176) 2024-09-18T12:03:42.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646e63b700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f6468097b70 0x7f646813adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58802/0 (socket says 172.21.15.112:58802) 2024-09-18T12:03:42.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646de3a700 1 -- 172.21.15.112:0/1286443706 learned_addr learned my addr 172.21.15.112:0/1286443706 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:42.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646de3a700 1 -- 172.21.15.112:0/1286443706 >> v1:172.21.15.112:6789/0 conn(0x7f64680a0380 legacy=0x7f64680a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646e63b700 1 -- 172.21.15.112:0/1286443706 >> v1:172.21.15.112:6790/0 conn(0x7f6468097b70 legacy=0x7f646813adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646de3a700 1 -- 172.21.15.112:0/1286443706 >> v1:172.21.15.112:6789/0 conn(0x7f64680a0380 legacy=0x7f64680a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-18T12:03:42.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646de3a700 1 -- 172.21.15.112:0/1286443706 >> v1:172.21.15.112:6789/0 conn(0x7f64680a0380 legacy=0x7f64680a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:03:42.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646de3a700 1 --1- 172.21.15.112:0/1286443706 >> v1:172.21.15.112:6789/0 conn(0x7f64680a0380 0x7f64680a0750 :-1 s=OPENED pgs=731 cs=1 l=1).handle_message read tag failed 2024-09-18T12:03:42.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646de3a700 1 --1- 172.21.15.112:0/1286443706 >> v1:172.21.15.112:6789/0 conn(0x7f64680a0380 0x7f64680a0750 :-1 s=OPENED pgs=731 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646d639700 1 -- 172.21.15.112:0/1286443706 >> v1:172.21.15.129:6789/0 conn(0x7f64680a0e60 legacy=0x7f6468137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.165+0000 7f646ce38700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6789/0 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18228150 0 0) 0x7f6468096dd0 con 0x7f6468097b70 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient(hunting): my global_id is 154292 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6454003570 con 0x7f6468097b70 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239983172 0 0) 0x7f64680a1550 con 0x7f64680a0e60 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient(hunting): my global_id is 144379 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6468096dd0 con 0x7f64680a0e60 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1679225749 0 0) 0x7f6454003570 con 0x7f6468097b70 2024-09-18T12:03:42.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 >> v1:172.21.15.129:6789/0 conn(0x7f64680a0e60 legacy=0x7f6468137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 >> v1:172.21.15.112:6789/0 conn(0x7f64680a0380 legacy=0x7f64680a0750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: _finish_hunting 0 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 monclient: found mon.noname-b 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64680a1e40 con 0x7f6468097b70 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: _finish_auth 0 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 5 monclient: authenticate success, global_id 154292 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6460001cc0 con 0x7f6468097b70 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: dump: 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:42.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6460001f50 con 0x7f6468097b70 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 1 -- 172.21.15.112:0/1286443706 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6460004d50 con 0x7f6468097b70 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 4 set_mon_vals no callback set 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ce38700 10 monclient: dump: 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.176 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.177 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.177 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.177 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 10 monclient: shutdown 2024-09-18T12:03:42.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:42.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 1 -- 172.21.15.112:0/1286443706 >> v1:172.21.15.112:6790/0 conn(0x7f6468097b70 legacy=0x7f646813adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 1 -- 172.21.15.112:0/1286443706 shutdown_connections 2024-09-18T12:03:42.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 1 -- 172.21.15.112:0/1286443706 >> 172.21.15.112:0/1286443706 conn(0x7f6468067680 msgr2=0x7f6468067a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:42.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 1 -- 172.21.15.112:0/1286443706 shutdown_connections 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 1 -- 172.21.15.112:0/1286443706 wait complete. 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 10 monclient: build_initial_monmap 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.169+0000 7f646ee3c700 10 monclient: monmap: 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:42.184 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 librados: starting msgr at 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 librados: starting objecter 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 Processor -- start 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 -- start start 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 librados: setting wanted keys 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 librados: calling monclient init 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 10 monclient: init 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 10 monclient(hunting): picked mon.noname-b con 0x7f6468097b70 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 10 monclient(hunting): picked mon.noname-c con 0x7f64680a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 10 monclient(hunting): picked mon.noname-a con 0x7f64680a0e60 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646811e700 con 0x7f64680a0e60 2024-09-18T12:03:42.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6468120b20 con 0x7f6468097b70 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646d639700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f64680a0380 0x7f64681418d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58806/0 (socket says 172.21.15.112:58806) 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646d639700 1 -- 172.21.15.112:0/434430100 learned_addr learned my addr 172.21.15.112:0/434430100 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6468120840 con 0x7f64680a0380 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.173+0000 7f646ee3c700 10 monclient(hunting): authenticate will time out at 1994.018053s 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646e63b700 1 -- 172.21.15.112:0/434430100 >> v1:172.21.15.112:6789/0 conn(0x7f64680a0e60 legacy=0x7f6468149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.?.objecter ms_handle_connect 0x7f64680a0e60 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.?.objecter ms_handle_connect 0x7f64680a0380 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646d639700 1 -- 172.21.15.112:0/434430100 >> v1:172.21.15.112:6790/0 conn(0x7f64680a0380 legacy=0x7f64681418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646de3a700 1 -- 172.21.15.112:0/434430100 >> v1:172.21.15.129:6789/0 conn(0x7f6468097b70 legacy=0x7f6468132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.?.objecter ms_handle_connect 0x7f6468097b70 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120255674 0 0) 0x7f6468120840 con 0x7f64680a0380 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): my global_id is 154295 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f644c004f10 con 0x7f64680a0380 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027273772 0 0) 0x7f646811e700 con 0x7f64680a0e60 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): my global_id is 164199 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6468120840 con 0x7f64680a0e60 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003108655 0 0) 0x7f6468120b20 con 0x7f6468097b70 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): my global_id is 144382 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646811e700 con 0x7f6468097b70 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4093201474 0 0) 0x7f644c004f10 con 0x7f64680a0380 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6468120b20 con 0x7f64680a0380 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6458001d20 con 0x7f64680a0380 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): dump: 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.188 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 519993905 0 0) 0x7f6468120840 con 0x7f64680a0e60 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f644c004f10 con 0x7f64680a0e60 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f6460004bb0 con 0x7f64680a0e60 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient(hunting): dump: 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.189 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2033332183 0 0) 0x7f646811e700 con 0x7f6468097b70 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6468120840 con 0x7f6468097b70 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3299055115 0 0) 0x7f6468120b20 con 0x7f64680a0380 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 >> v1:172.21.15.129:6789/0 conn(0x7f6468097b70 legacy=0x7f6468132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 >> v1:172.21.15.112:6789/0 conn(0x7f64680a0e60 legacy=0x7f6468149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient: _finish_hunting 0 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 monclient: found mon.c 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6468120e00 con 0x7f64680a0380 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient: _finish_auth 0 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 5 monclient: authenticate success, global_id 154295 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 10 monclient: _renew_subs 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646811f950 con 0x7f64680a0380 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 10 client.154295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 10 monclient: _renew_subs 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6468132fc0 con 0x7f64680a0380 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 1 librados: init done 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64580022f0 con 0x7f64680a0380 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f64580024a0 con 0x7f64680a0380 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645ffff700 4 set_mon_vals no callback set 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient: dump: 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.192 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 10 client.154295.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 15 client.154295.objecter _session_command_op_assign -1 1 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 20 client.154295.objecter _get_session osd=-1 returning homeless 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 10 client.154295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.112:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f64480028c0 con 0x7f64680a0380 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f6458002700 con 0x7f64680a0380 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f6458043a80 con 0x7f64680a0380 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.154295.objecter ms_dispatch 0x7f6468000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:42.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 3 client.154295.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 3 client.154295.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.154295.objecter checking command 1 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 20 client.154295.objecter _get_session s=0x7f644c040240 osd=3 3 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 20 client.154295.objecter put_session s=0x7f644c040240 osd=3 3 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 15 client.154295.objecter _session_command_op_remove -1 1 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 20 client.154295.objecter _get_session s=0x7f644c040240 osd=3 3 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 20 client.154295.objecter get_session s=0x7f644c040240 osd=3 3 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 15 client.154295.objecter _session_command_op_assign 3 1 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 20 client.154295.objecter put_session s=0x7f644c040240 osd=3 4 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 client.154295.objecter _send_command 1 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.129:6804/21049 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f644c0445f0 con 0x7f644c0408f0 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 20 client.154295.objecter dump_active .. 0 homeless 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_get_version_reply(handle=1 version=699) v2 ==== 24+0+0 (unknown 757610054 0 0) 0x7f6458043eb0 con 0x7f64680a0380 2024-09-18T12:03:42.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.177+0000 7f645dffb700 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-18T12:03:42.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.181+0000 7f646de3a700 10 monclient: get_auth_request con 0x7f644c038650 auth_method 0 2024-09-18T12:03:42.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.181+0000 7f646e63b700 10 monclient: get_auth_request con 0x7f644c0408f0 auth_method 0 2024-09-18T12:03:42.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.181+0000 7f645dffb700 10 client.154295.objecter ms_handle_connect 0x7f644c038650 2024-09-18T12:03:42.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.185+0000 7f645dffb700 10 client.154295.objecter ms_handle_connect 0x7f644c0408f0 2024-09-18T12:03:42.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.185+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== osd.3 v1:172.21.15.129:6804/21049 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f644c0445f0 con 0x7f644c0408f0 2024-09-18T12:03:42.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.185+0000 7f645dffb700 10 client.154295.objecter ms_dispatch 0x7f6468000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T12:03:42.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.185+0000 7f645dffb700 10 client.154295.objecter _finish_command 1 = system:0 2024-09-18T12:03:42.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.185+0000 7f645dffb700 20 client.154295.objecter put_session s=0x7f644c040240 osd=3 4 2024-09-18T12:03:42.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.185+0000 7f645dffb700 15 client.154295.objecter _session_command_op_remove 3 1 2024-09-18T12:03:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 10 client.154295.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-18T12:03:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 15 client.154295.objecter _session_command_op_assign -1 2 2024-09-18T12:03:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 20 client.154295.objecter _get_session s=0x7f644c040240 osd=3 3 2024-09-18T12:03:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 20 client.154295.objecter put_session s=0x7f644c040240 osd=3 3 2024-09-18T12:03:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 20 client.154295.objecter _get_session s=0x7f644c040240 osd=3 3 2024-09-18T12:03:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 15 client.154295.objecter _session_command_op_remove -1 2 2024-09-18T12:03:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 20 client.154295.objecter get_session s=0x7f644c040240 osd=3 3 2024-09-18T12:03:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 15 client.154295.objecter _session_command_op_assign 3 2 2024-09-18T12:03:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 20 client.154295.objecter put_session s=0x7f644c040240 osd=3 4 2024-09-18T12:03:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 10 client.154295.objecter _send_command 2 2024-09-18T12:03:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 --> v1:172.21.15.129:6804/21049 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f64480011f0 con 0x7f644c0408f0 2024-09-18T12:03:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f645dffb700 1 -- 172.21.15.112:0/434430100 <== osd.3 v1:172.21.15.129:6804/21049 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f64480011f0 con 0x7f644c0408f0 2024-09-18T12:03:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f645dffb700 10 client.154295.objecter ms_dispatch 0x7f6468000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T12:03:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f645dffb700 10 client.154295.objecter _finish_command 2 = system:0 2024-09-18T12:03:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f645dffb700 20 client.154295.objecter put_session s=0x7f644c040240 osd=3 4 2024-09-18T12:03:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f645dffb700 15 client.154295.objecter _session_command_op_remove 3 2 2024-09-18T12:03:42.209 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:03:42.209 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 10 librados: watch_flush enter 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 10 librados: watch_flush exit 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 10 client.154295.objecter close_session for osd.3 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 >> v1:172.21.15.129:6804/21049 conn(0x7f644c0408f0 legacy=0x7f644c042d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 20 client.154295.objecter put_session s=0x7f644c040240 osd=3 1 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 20 client.154295.objecter shutdown clearing up homeless session... 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 10 client.154295.objecter successfully canceled tick 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 >> v1:172.21.15.129:6812/20806 conn(0x7f644c038650 legacy=0x7f644c03ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 10 monclient: shutdown 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 >> v1:172.21.15.112:6790/0 conn(0x7f64680a0380 legacy=0x7f64681418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646e63b700 1 -- 172.21.15.112:0/434430100 reap_dead start 2024-09-18T12:03:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 shutdown_connections 2024-09-18T12:03:42.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 >> 172.21.15.112:0/434430100 conn(0x7f6468067680 msgr2=0x7f64680a4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:42.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 shutdown_connections 2024-09-18T12:03:42.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 -- 172.21.15.112:0/434430100 wait complete. 2024-09-18T12:03:42.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.205+0000 7f646ee3c700 1 librados: shutdown 2024-09-18T12:03:42.221 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-18T12:03:42.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.329+0000 7f090f43e700 10 monclient: get_monmap_and_config 2024-09-18T12:03:42.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.329+0000 7f090f43e700 10 monclient: build_initial_monmap 2024-09-18T12:03:42.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.329+0000 7f090f43e700 10 monclient: monmap: 2024-09-18T12:03:42.334 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:42.335526+0000 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:42.335526+0000 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:42.335 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.329+0000 7f090f43e700 1 Processor -- start 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.329+0000 7f090f43e700 1 -- start start 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.329+0000 7f090f43e700 10 monclient: init 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 10 monclient(hunting): picked mon.noname-b con 0x7f09080a20c0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 10 monclient(hunting): picked mon.noname-c con 0x7f090809e370 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 10 monclient(hunting): picked mon.noname-a con 0x7f0908134750 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090812b940 con 0x7f0908134750 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090806eea0 con 0x7f090809e370 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090806f180 con 0x7f09080a20c0 2024-09-18T12:03:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:42.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090f43e700 10 monclient(hunting): authenticate will time out at 1994.177375s 2024-09-18T12:03:42.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090e43c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f09080a20c0 0x7f090812b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58818/0 (socket says 172.21.15.112:58818) 2024-09-18T12:03:42.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090e43c700 1 -- 172.21.15.112:0/841685380 learned_addr learned my addr 172.21.15.112:0/841685380 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090e43c700 1 -- 172.21.15.112:0/841685380 >> v1:172.21.15.112:6790/0 conn(0x7f09080a20c0 legacy=0x7f090812b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090ec3d700 1 -- 172.21.15.112:0/841685380 >> v1:172.21.15.112:6789/0 conn(0x7f0908134750 legacy=0x7f0908136b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245384066 0 0) 0x7f090806f180 con 0x7f09080a20c0 2024-09-18T12:03:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090d43a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090d43a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090d43a700 10 monclient(hunting): my global_id is 154298 2024-09-18T12:03:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090dc3b700 1 -- 172.21.15.112:0/841685380 >> v1:172.21.15.129:6789/0 conn(0x7f090809e370 legacy=0x7f090809e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.333+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f4003570 con 0x7f09080a20c0 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557316855 0 0) 0x7f090812b940 con 0x7f0908134750 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient(hunting): my global_id is 164202 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090806f180 con 0x7f0908134750 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597606870 0 0) 0x7f090806eea0 con 0x7f090809e370 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient(hunting): my global_id is 144385 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090812b940 con 0x7f090809e370 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3179359247 0 0) 0x7f08f4003570 con 0x7f09080a20c0 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 >> v1:172.21.15.129:6789/0 conn(0x7f090809e370 legacy=0x7f090809e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 >> v1:172.21.15.112:6789/0 conn(0x7f0908134750 legacy=0x7f0908136b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: _finish_hunting 0 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 monclient: found mon.noname-b 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0908096b00 con 0x7f09080a20c0 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: _finish_auth 0 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08fc001a00 con 0x7f09080a20c0 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: dump: 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 5 monclient: authenticate success, global_id 154298 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08fc001be0 con 0x7f09080a20c0 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 1 -- 172.21.15.112:0/841685380 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08fc004a90 con 0x7f09080a20c0 2024-09-18T12:03:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 4 set_mon_vals no callback set 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090d43a700 10 monclient: dump: 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.345 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.345 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.345 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.345 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 10 monclient: shutdown 2024-09-18T12:03:42.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:42.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 1 -- 172.21.15.112:0/841685380 >> v1:172.21.15.112:6790/0 conn(0x7f09080a20c0 legacy=0x7f090812b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.350 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 1 -- 172.21.15.112:0/841685380 shutdown_connections 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 1 -- 172.21.15.112:0/841685380 >> 172.21.15.112:0/841685380 conn(0x7f0908067da0 msgr2=0x7f09080a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 1 -- 172.21.15.112:0/841685380 shutdown_connections 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 1 -- 172.21.15.112:0/841685380 wait complete. 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 10 monclient: build_initial_monmap 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 10 monclient: monmap: 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 1 librados: starting msgr at 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.337+0000 7f090f43e700 1 librados: starting objecter 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 1 Processor -- start 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 1 -- start start 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 1 librados: setting wanted keys 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 1 librados: calling monclient init 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 10 monclient: init 2024-09-18T12:03:42.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 10 monclient(hunting): picked mon.noname-a con 0x7f090809e370 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 10 monclient(hunting): picked mon.noname-b con 0x7f09080a20c0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 10 monclient(hunting): picked mon.noname-c con 0x7f0908134750 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090809b530 con 0x7f090809e370 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090813dec0 con 0x7f09080a20c0 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090809ce40 con 0x7f0908134750 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090f43e700 10 monclient(hunting): authenticate will time out at 1994.185606s 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090e43c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f090809e370 0x7f090813d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46200/0 (socket says 172.21.15.112:46200) 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090e43c700 1 -- 172.21.15.112:0/1603821935 learned_addr learned my addr 172.21.15.112:0/1603821935 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.341+0000 7f090e43c700 1 -- 172.21.15.112:0/1603821935 >> v1:172.21.15.112:6789/0 conn(0x7f090809e370 legacy=0x7f090813d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090ec3d700 1 -- 172.21.15.112:0/1603821935 >> v1:172.21.15.112:6790/0 conn(0x7f0908134750 legacy=0x7f0908145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 client.?.objecter ms_handle_connect 0x7f090809e370 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 client.?.objecter ms_handle_connect 0x7f0908134750 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090dc3b700 1 -- 172.21.15.112:0/1603821935 >> v1:172.21.15.129:6789/0 conn(0x7f09080a20c0 legacy=0x7f090813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789278570 0 0) 0x7f090809b530 con 0x7f090809e370 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): my global_id is 164205 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ec004f10 con 0x7f090809e370 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 client.?.objecter ms_handle_connect 0x7f09080a20c0 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573676792 0 0) 0x7f090809ce40 con 0x7f0908134750 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): my global_id is 154301 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090809b530 con 0x7f0908134750 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173025414 0 0) 0x7f090813dec0 con 0x7f09080a20c0 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): my global_id is 144388 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090809ce40 con 0x7f09080a20c0 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 168389870 0 0) 0x7f090809b530 con 0x7f0908134750 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f090813dec0 con 0x7f0908134750 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0900002000 con 0x7f0908134750 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:42.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): dump: 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 713878915 0 0) 0x7f08ec004f10 con 0x7f090809e370 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f090809b530 con 0x7f090809e370 2024-09-18T12:03:42.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f08fc004830 con 0x7f090809e370 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient(hunting): dump: 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.357 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 988116395 0 0) 0x7f090813dec0 con 0x7f0908134750 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 >> v1:172.21.15.129:6789/0 conn(0x7f09080a20c0 legacy=0x7f090813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 >> v1:172.21.15.112:6789/0 conn(0x7f090809e370 legacy=0x7f090813d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 monclient: found mon.c 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090809d120 con 0x7f0908134750 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient: _finish_auth 0 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 5 monclient: authenticate success, global_id 154301 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 10 monclient: _renew_subs 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090809d400 con 0x7f0908134750 2024-09-18T12:03:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 10 client.154301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 10 monclient: _renew_subs 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f090809ade0 con 0x7f0908134750 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 1 librados: init done 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0900002530 con 0x7f0908134750 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f09000028c0 con 0x7f0908134750 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fbfff700 4 set_mon_vals no callback set 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 monclient: dump: 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f0900002a20 con 0x7f0908134750 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f00050b0 con 0x7f0908134750 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f0900043da0 con 0x7f0908134750 2024-09-18T12:03:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 10 client.154301.objecter ms_dispatch 0x7f0908000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 3 client.154301.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 3 client.154301.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f08fa7fc700 20 client.154301.objecter dump_active .. 0 homeless 2024-09-18T12:03:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.345+0000 7f090e43c700 10 monclient: get_auth_request con 0x7f08ec038650 auth_method 0 2024-09-18T12:03:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.349+0000 7f08fa7fc700 10 client.154301.objecter ms_handle_connect 0x7f08ec038650 2024-09-18T12:03:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.353+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f09000030f0 con 0x7f0908134750 2024-09-18T12:03:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.353+0000 7f08fa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.353+0000 7f08fa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:42.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.517+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f08f0000b90 con 0x7f08ec038650 2024-09-18T12:03:42.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f08fa7fc700 1 -- 172.21.15.112:0/1603821935 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (unknown 3526570807 0 0) 0x7f08f0000b90 con 0x7f08ec038650 2024-09-18T12:03:42.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 10 librados: watch_flush enter 2024-09-18T12:03:42.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 10 librados: watch_flush exit 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 20 client.154301.objecter shutdown clearing up homeless session... 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 10 client.154301.objecter successfully canceled tick 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 >> v1:172.21.15.129:6812/20806 conn(0x7f08ec038650 legacy=0x7f08ec03ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 10 monclient: shutdown 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 >> v1:172.21.15.112:6790/0 conn(0x7f0908134750 legacy=0x7f0908145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 shutdown_connections 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 >> 172.21.15.112:0/1603821935 conn(0x7f0908067da0 msgr2=0x7f0908126eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 shutdown_connections 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 1 -- 172.21.15.112:0/1603821935 wait complete. 2024-09-18T12:03:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.521+0000 7f090f43e700 1 librados: shutdown 2024-09-18T12:03:42.527 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 3 to deep-scrub 2024-09-18T12:03:42.536 INFO:tasks.ceph:Scrubbing osd.4 2024-09-18T12:03:42.537 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:03:42.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient: get_monmap_and_config 2024-09-18T12:03:42.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient: build_initial_monmap 2024-09-18T12:03:42.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient: monmap: 2024-09-18T12:03:42.648 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:42.649685+0000 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:42.649685+0000 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:42.649 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 1 Processor -- start 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 1 -- start start 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient: init 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient(hunting): picked mon.noname-c con 0x7ffa400a0380 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient(hunting): picked mon.noname-a con 0x7ffa400a0e60 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient(hunting): picked mon.noname-b con 0x7ffa40097b70 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa40097f90 con 0x7ffa400a0e60 2024-09-18T12:03:42.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa400a1550 con 0x7ffa400a0380 2024-09-18T12:03:42.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa40096dd0 con 0x7ffa40097b70 2024-09-18T12:03:42.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:42.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa459ac700 10 monclient(hunting): authenticate will time out at 1994.491658s 2024-09-18T12:03:42.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa451ab700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ffa40097b70 0x7ffa4013adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58832/0 (socket says 172.21.15.112:58832) 2024-09-18T12:03:42.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa3ffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ffa400a0e60 0x7ffa40137690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46208/0 (socket says 172.21.15.112:46208) 2024-09-18T12:03:42.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.645+0000 7ffa451ab700 1 -- 172.21.15.112:0/1754465466 learned_addr learned my addr 172.21.15.112:0/1754465466 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:42.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa451ab700 1 -- 172.21.15.112:0/1754465466 >> v1:172.21.15.112:6790/0 conn(0x7ffa40097b70 legacy=0x7ffa4013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3ffff700 1 -- 172.21.15.112:0/1754465466 >> v1:172.21.15.112:6789/0 conn(0x7ffa400a0e60 legacy=0x7ffa40137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa449aa700 1 -- 172.21.15.112:0/1754465466 >> v1:172.21.15.129:6789/0 conn(0x7ffa400a0380 legacy=0x7ffa400a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227768012 0 0) 0x7ffa40096dd0 con 0x7ffa40097b70 2024-09-18T12:03:42.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): my global_id is 154304 2024-09-18T12:03:42.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa28003570 con 0x7ffa40097b70 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139688063 0 0) 0x7ffa40097f90 con 0x7ffa400a0e60 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): my global_id is 164208 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa40096dd0 con 0x7ffa400a0e60 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536588108 0 0) 0x7ffa400a1550 con 0x7ffa400a0380 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient(hunting): my global_id is 144391 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa40097f90 con 0x7ffa400a0380 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3164555270 0 0) 0x7ffa28003570 con 0x7ffa40097b70 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 >> v1:172.21.15.129:6789/0 conn(0x7ffa400a0380 legacy=0x7ffa400a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 >> v1:172.21.15.112:6789/0 conn(0x7ffa400a0e60 legacy=0x7ffa40137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.656 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 monclient: found mon.noname-b 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa400a1e40 con 0x7ffa40097b70 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: _finish_auth 0 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa38001cc0 con 0x7ffa40097b70 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: dump: 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.657 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 5 monclient: authenticate success, global_id 154304 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa38001f50 con 0x7ffa40097b70 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 4 set_mon_vals no callback set 2024-09-18T12:03:42.658 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 1 -- 172.21.15.112:0/1754465466 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa38004e10 con 0x7ffa40097b70 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa3f7fe700 10 monclient: dump: 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 10 monclient: shutdown 2024-09-18T12:03:42.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:42.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 1 -- 172.21.15.112:0/1754465466 >> v1:172.21.15.112:6790/0 conn(0x7ffa40097b70 legacy=0x7ffa4013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 1 -- 172.21.15.112:0/1754465466 shutdown_connections 2024-09-18T12:03:42.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 1 -- 172.21.15.112:0/1754465466 >> 172.21.15.112:0/1754465466 conn(0x7ffa40067680 msgr2=0x7ffa40067a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:42.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 1 -- 172.21.15.112:0/1754465466 shutdown_connections 2024-09-18T12:03:42.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.649+0000 7ffa459ac700 1 -- 172.21.15.112:0/1754465466 wait complete. 2024-09-18T12:03:42.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient: build_initial_monmap 2024-09-18T12:03:42.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient: monmap: 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 librados: starting msgr at 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 librados: starting objecter 2024-09-18T12:03:42.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 Processor -- start 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 -- start start 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 librados: setting wanted keys 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 librados: calling monclient init 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient: init 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient(hunting): picked mon.noname-a con 0x7ffa40097b70 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient(hunting): picked mon.noname-c con 0x7ffa400a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient(hunting): picked mon.noname-b con 0x7ffa400a0e60 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa449aa700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ffa40097b70 0x7ffa40141720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46214/0 (socket says 172.21.15.112:46214) 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4011e0f0 con 0x7ffa40097b70 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa449aa700 1 -- 172.21.15.112:0/393756556 learned_addr learned my addr 172.21.15.112:0/393756556 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:42.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa400986d0 con 0x7ffa400a0e60 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa401425e0 con 0x7ffa400a0380 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.653+0000 7ffa459ac700 10 monclient(hunting): authenticate will time out at 1994.499006s 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa449aa700 1 -- 172.21.15.112:0/393756556 >> v1:172.21.15.112:6789/0 conn(0x7ffa40097b70 legacy=0x7ffa40141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 client.?.objecter ms_handle_connect 0x7ffa40097b70 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 client.?.objecter ms_handle_connect 0x7ffa400a0380 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3ffff700 1 -- 172.21.15.112:0/393756556 >> v1:172.21.15.112:6790/0 conn(0x7ffa400a0380 legacy=0x7ffa40132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950987705 0 0) 0x7ffa4011e0f0 con 0x7ffa40097b70 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): my global_id is 164211 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa2c004f10 con 0x7ffa40097b70 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 client.?.objecter ms_handle_connect 0x7ffa400a0e60 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa451ab700 1 -- 172.21.15.112:0/393756556 >> v1:172.21.15.129:6789/0 conn(0x7ffa400a0e60 legacy=0x7ffa40149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336550569 0 0) 0x7ffa401425e0 con 0x7ffa400a0380 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): my global_id is 154307 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4011e0f0 con 0x7ffa400a0380 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212167896 0 0) 0x7ffa400986d0 con 0x7ffa400a0e60 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): my global_id is 144394 2024-09-18T12:03:42.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa401425e0 con 0x7ffa400a0e60 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 345005096 0 0) 0x7ffa4011e0f0 con 0x7ffa400a0380 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa400986d0 con 0x7ffa400a0380 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa30001cb0 con 0x7ffa400a0380 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): dump: 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.670 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554526654 0 0) 0x7ffa2c004f10 con 0x7ffa40097b70 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa4011e0f0 con 0x7ffa40097b70 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa34001f00 con 0x7ffa40097b70 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient(hunting): dump: 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.671 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2997746006 0 0) 0x7ffa400986d0 con 0x7ffa400a0380 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 >> v1:172.21.15.129:6789/0 conn(0x7ffa400a0e60 legacy=0x7ffa40149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 >> v1:172.21.15.112:6789/0 conn(0x7ffa40097b70 legacy=0x7ffa40141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient: _finish_hunting 0 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 monclient: found mon.c 2024-09-18T12:03:42.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa400a20f0 con 0x7ffa400a0380 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient: _finish_auth 0 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 5 monclient: authenticate success, global_id 154307 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 10 monclient: _renew_subs 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4011e740 con 0x7ffa400a0380 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 10 client.154307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 10 monclient: _renew_subs 2024-09-18T12:03:42.673 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa40132c20 con 0x7ffa400a0380 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 1 librados: init done 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa30004ae0 con 0x7ffa400a0380 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ffa30004df0 con 0x7ffa400a0380 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3effd700 4 set_mon_vals no callback set 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 10 monclient: dump: 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 10 client.154307.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:42.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 15 client.154307.objecter _session_command_op_assign -1 1 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 20 client.154307.objecter _get_session osd=-1 returning homeless 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 10 client.154307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.112:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa200028c0 con 0x7ffa400a0380 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.657+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ffa30004fd0 con 0x7ffa400a0380 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7ffa30043ca0 con 0x7ffa400a0380 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 10 client.154307.objecter ms_dispatch 0x7ffa40000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 3 client.154307.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 3 client.154307.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 10 client.154307.objecter checking command 1 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 20 client.154307.objecter _get_session s=0x7ffa2c03ffb0 osd=4 3 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 20 client.154307.objecter put_session s=0x7ffa2c03ffb0 osd=4 3 2024-09-18T12:03:42.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 15 client.154307.objecter _session_command_op_remove -1 1 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 20 client.154307.objecter _get_session s=0x7ffa2c03ffb0 osd=4 3 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 20 client.154307.objecter get_session s=0x7ffa2c03ffb0 osd=4 3 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 15 client.154307.objecter _session_command_op_assign 4 1 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 20 client.154307.objecter put_session s=0x7ffa2c03ffb0 osd=4 4 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 10 client.154307.objecter _send_command 1 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.129:6808/21047 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa2c044360 con 0x7ffa2c040660 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 20 client.154307.objecter dump_active .. 0 homeless 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_get_version_reply(handle=1 version=699) v2 ==== 24+0+0 (unknown 757610054 0 0) 0x7ffa300440d0 con 0x7ffa400a0380 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa449aa700 10 monclient: get_auth_request con 0x7ffa2c0383f0 auth_method 0 2024-09-18T12:03:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa451ab700 10 monclient: get_auth_request con 0x7ffa2c040660 auth_method 0 2024-09-18T12:03:42.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.661+0000 7ffa3cff9700 10 client.154307.objecter ms_handle_connect 0x7ffa2c0383f0 2024-09-18T12:03:42.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.665+0000 7ffa3cff9700 10 client.154307.objecter ms_handle_connect 0x7ffa2c040660 2024-09-18T12:03:42.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.665+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== osd.4 v1:172.21.15.129:6808/21047 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ffa2c044360 con 0x7ffa2c040660 2024-09-18T12:03:42.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.665+0000 7ffa3cff9700 10 client.154307.objecter ms_dispatch 0x7ffa40000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T12:03:42.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.665+0000 7ffa3cff9700 10 client.154307.objecter _finish_command 1 = system:0 2024-09-18T12:03:42.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.665+0000 7ffa3cff9700 20 client.154307.objecter put_session s=0x7ffa2c03ffb0 osd=4 4 2024-09-18T12:03:42.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.665+0000 7ffa3cff9700 15 client.154307.objecter _session_command_op_remove 4 1 2024-09-18T12:03:42.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 10 client.154307.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-18T12:03:42.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 15 client.154307.objecter _session_command_op_assign -1 2 2024-09-18T12:03:42.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 20 client.154307.objecter _get_session s=0x7ffa2c03ffb0 osd=4 3 2024-09-18T12:03:42.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 20 client.154307.objecter put_session s=0x7ffa2c03ffb0 osd=4 3 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 20 client.154307.objecter _get_session s=0x7ffa2c03ffb0 osd=4 3 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 15 client.154307.objecter _session_command_op_remove -1 2 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 20 client.154307.objecter get_session s=0x7ffa2c03ffb0 osd=4 3 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 15 client.154307.objecter _session_command_op_assign 4 2 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 20 client.154307.objecter put_session s=0x7ffa2c03ffb0 osd=4 4 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 10 client.154307.objecter _send_command 2 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.685+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 --> v1:172.21.15.129:6808/21047 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ffa200011f0 con 0x7ffa2c040660 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa3cff9700 1 -- 172.21.15.112:0/393756556 <== osd.4 v1:172.21.15.129:6808/21047 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7ffa200011f0 con 0x7ffa2c040660 2024-09-18T12:03:42.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa3cff9700 10 client.154307.objecter ms_dispatch 0x7ffa40000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T12:03:42.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa3cff9700 10 client.154307.objecter _finish_command 2 = system:0 2024-09-18T12:03:42.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa3cff9700 20 client.154307.objecter put_session s=0x7ffa2c03ffb0 osd=4 4 2024-09-18T12:03:42.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa3cff9700 15 client.154307.objecter _session_command_op_remove 4 2 2024-09-18T12:03:42.692 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:42.692 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:03:42.692 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 10 librados: watch_flush enter 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 10 librados: watch_flush exit 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 10 client.154307.objecter close_session for osd.4 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 >> v1:172.21.15.129:6808/21047 conn(0x7ffa2c040660 legacy=0x7ffa2c042af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 20 client.154307.objecter put_session s=0x7ffa2c03ffb0 osd=4 1 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 20 client.154307.objecter shutdown clearing up homeless session... 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 10 client.154307.objecter successfully canceled tick 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 >> v1:172.21.15.129:6812/20806 conn(0x7ffa2c0383f0 legacy=0x7ffa2c03a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 10 monclient: shutdown 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:42.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 >> v1:172.21.15.112:6790/0 conn(0x7ffa400a0380 legacy=0x7ffa40132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa451ab700 1 -- 172.21.15.112:0/393756556 reap_dead start 2024-09-18T12:03:42.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 shutdown_connections 2024-09-18T12:03:42.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 >> 172.21.15.112:0/393756556 conn(0x7ffa40067680 msgr2=0x7ffa400989e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:42.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 shutdown_connections 2024-09-18T12:03:42.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 1 -- 172.21.15.112:0/393756556 wait complete. 2024-09-18T12:03:42.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.689+0000 7ffa459ac700 1 librados: shutdown 2024-09-18T12:03:42.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient: get_monmap_and_config 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient: build_initial_monmap 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient: monmap: 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:42.818002+0000 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:42.818002+0000 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:42.817 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:42.818 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 1 Processor -- start 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 1 -- start start 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient: init 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient(hunting): picked mon.noname-a con 0x7f03240a2200 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient(hunting): picked mon.noname-c con 0x7f03240a01e0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient(hunting): picked mon.noname-b con 0x7f0324138600 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032406e890 con 0x7f03240a2200 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240a16e0 con 0x7f03240a01e0 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240a25d0 con 0x7f0324138600 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f032ac35700 10 monclient(hunting): authenticate will time out at 1994.658815s 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f03289d1700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f03240a2200 0x7f032409ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46216/0 (socket says 172.21.15.112:46216) 2024-09-18T12:03:42.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f03289d1700 1 -- 172.21.15.112:0/2413220275 learned_addr learned my addr 172.21.15.112:0/2413220275 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:42.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.813+0000 7f03291d2700 1 --1- 172.21.15.112:0/2413220275 >> v1:172.21.15.112:6790/0 conn(0x7f0324138600 0x7f032413a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58842/0 (socket says 172.21.15.112:58842) 2024-09-18T12:03:42.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03291d2700 1 -- 172.21.15.112:0/2413220275 >> v1:172.21.15.112:6790/0 conn(0x7f0324138600 legacy=0x7f032413a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03289d1700 1 -- 172.21.15.112:0/2413220275 >> v1:172.21.15.112:6789/0 conn(0x7f03240a2200 legacy=0x7f032409ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f0323fff700 1 -- 172.21.15.112:0/2413220275 >> v1:172.21.15.129:6789/0 conn(0x7f03240a01e0 legacy=0x7f032409b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109481455 0 0) 0x7f03240a25d0 con 0x7f0324138600 2024-09-18T12:03:42.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): my global_id is 154310 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0314003570 con 0x7f0324138600 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964729123 0 0) 0x7f0310001450 con 0x7f03240a2200 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): my global_id is 164214 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03240a25d0 con 0x7f03240a2200 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454399592 0 0) 0x7f03240a16e0 con 0x7f03240a01e0 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient(hunting): my global_id is 144397 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310001450 con 0x7f03240a01e0 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3117373083 0 0) 0x7f03240a25d0 con 0x7f03240a2200 2024-09-18T12:03:42.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 >> v1:172.21.15.112:6790/0 conn(0x7f0324138600 legacy=0x7f032413a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 >> v1:172.21.15.129:6789/0 conn(0x7f03240a01e0 legacy=0x7f032409b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: _finish_hunting 0 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 monclient: found mon.noname-a 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032409bc90 con 0x7f03240a2200 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: _finish_auth 0 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 5 monclient: authenticate success, global_id 164214 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0310001910 con 0x7f03240a2200 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: dump: 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0310001ba0 con 0x7f03240a2200 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 4 set_mon_vals no callback set 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 1 -- 172.21.15.112:0/2413220275 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0310004a90 con 0x7f03240a2200 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f03237fe700 10 monclient: dump: 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.827 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 10 monclient: shutdown 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 1 -- 172.21.15.112:0/2413220275 >> v1:172.21.15.112:6789/0 conn(0x7f03240a2200 legacy=0x7f032409ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 1 -- 172.21.15.112:0/2413220275 shutdown_connections 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 1 -- 172.21.15.112:0/2413220275 >> 172.21.15.112:0/2413220275 conn(0x7f03240680d0 msgr2=0x7f03240a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 1 -- 172.21.15.112:0/2413220275 shutdown_connections 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.817+0000 7f032ac35700 1 -- 172.21.15.112:0/2413220275 wait complete. 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient: build_initial_monmap 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient: monmap: 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:42.828 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 librados: starting msgr at 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 librados: starting objecter 2024-09-18T12:03:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 Processor -- start 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 -- start start 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 librados: setting wanted keys 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 librados: calling monclient init 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient: init 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient(hunting): picked mon.noname-b con 0x7f03240a01e0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient(hunting): picked mon.noname-a con 0x7f03240a2200 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:42.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient(hunting): picked mon.noname-c con 0x7f0324138600 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032409ec90 con 0x7f03240a2200 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240a0f80 con 0x7f03240a01e0 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324131ac0 con 0x7f0324138600 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f032ac35700 10 monclient(hunting): authenticate will time out at 1994.665432s 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f0323fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f03240a2200 0x7f03241313b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46218/0 (socket says 172.21.15.112:46218) 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f0323fff700 1 -- 172.21.15.112:0/4096864181 learned_addr learned my addr 172.21.15.112:0/4096864181 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f0323fff700 1 -- 172.21.15.112:0/4096864181 >> v1:172.21.15.112:6789/0 conn(0x7f03240a2200 legacy=0x7f03241313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f0320ff9700 10 client.?.objecter ms_handle_connect 0x7f03240a2200 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f0320ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f0320ff9700 10 client.?.objecter ms_handle_connect 0x7f0324138600 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f0320ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.821+0000 7f03291d2700 1 -- 172.21.15.112:0/4096864181 >> v1:172.21.15.112:6790/0 conn(0x7f0324138600 legacy=0x7f03241497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 client.?.objecter ms_handle_connect 0x7f03240a01e0 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f03289d1700 1 -- 172.21.15.112:0/4096864181 >> v1:172.21.15.129:6789/0 conn(0x7f03240a01e0 legacy=0x7f0324141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776219628 0 0) 0x7f032409ec90 con 0x7f03240a2200 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): my global_id is 164217 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0308004f10 con 0x7f03240a2200 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304227798 0 0) 0x7f0324131ac0 con 0x7f0324138600 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): my global_id is 154313 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032409ec90 con 0x7f0324138600 2024-09-18T12:03:42.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269311524 0 0) 0x7f03240a0f80 con 0x7f03240a01e0 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): my global_id is 144400 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0324131ac0 con 0x7f03240a01e0 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553899542 0 0) 0x7f0308004f10 con 0x7f03240a2200 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03240a0f80 con 0x7f03240a2200 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0318001cf0 con 0x7f03240a2200 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): dump: 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.837 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3970101627 0 0) 0x7f032409ec90 con 0x7f0324138600 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0308004f10 con 0x7f0324138600 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f031c001d30 con 0x7f0324138600 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient(hunting): dump: 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2995727046 0 0) 0x7f0324131ac0 con 0x7f03240a01e0 2024-09-18T12:03:42.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f032409ec90 con 0x7f03240a01e0 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2375912647 0 0) 0x7f03240a0f80 con 0x7f03240a2200 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 >> v1:172.21.15.112:6790/0 conn(0x7f0324138600 legacy=0x7f03241497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 >> v1:172.21.15.129:6789/0 conn(0x7f03240a01e0 legacy=0x7f0324141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient: _finish_hunting 0 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 monclient: found mon.a 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324137420 con 0x7f03240a2200 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient: _finish_auth 0 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 5 monclient: authenticate success, global_id 164217 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 10 monclient: _renew_subs 2024-09-18T12:03:42.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0324137ae0 con 0x7f03240a2200 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 10 client.164217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 10 monclient: _renew_subs 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0318002300 con 0x7f03240a2200 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0324142200 con 0x7f03240a2200 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 1 librados: init done 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f0318002690 con 0x7f03240a2200 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:42.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 monclient: dump: 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f03227fc700 4 set_mon_vals no callback set 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f03180027f0 con 0x7f03240a2200 2024-09-18T12:03:42.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0324118580 con 0x7f03240a2200 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f0318043bb0 con 0x7f03240a2200 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 10 client.164217.objecter ms_dispatch 0x7f0324000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 3 client.164217.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 3 client.164217.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f0320ff9700 20 client.164217.objecter dump_active .. 0 homeless 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.825+0000 7f03289d1700 10 monclient: get_auth_request con 0x7f0308038670 auth_method 0 2024-09-18T12:03:42.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.829+0000 7f0320ff9700 10 client.164217.objecter ms_handle_connect 0x7f0308038670 2024-09-18T12:03:42.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.833+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f0318014a10 con 0x7f03240a2200 2024-09-18T12:03:42.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.833+0000 7f0320ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:42.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.833+0000 7f0320ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:43.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.997+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f032405f2e0 con 0x7f0308038670 2024-09-18T12:03:43.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:42.997+0000 7f0320ff9700 1 -- 172.21.15.112:0/4096864181 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (unknown 3782719169 0 0) 0x7f032405f2e0 con 0x7f0308038670 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 10 librados: watch_flush enter 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 10 librados: watch_flush exit 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 20 client.164217.objecter shutdown clearing up homeless session... 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 10 client.164217.objecter successfully canceled tick 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 >> v1:172.21.15.129:6812/20806 conn(0x7f0308038670 legacy=0x7f030803ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 10 monclient: shutdown 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 >> v1:172.21.15.112:6789/0 conn(0x7f03240a2200 legacy=0x7f03241313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 shutdown_connections 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 >> 172.21.15.112:0/4096864181 conn(0x7f03240680d0 msgr2=0x7f032406bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 shutdown_connections 2024-09-18T12:03:43.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 1 -- 172.21.15.112:0/4096864181 wait complete. 2024-09-18T12:03:43.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.001+0000 7f032ac35700 1 librados: shutdown 2024-09-18T12:03:43.005 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 4 to deep-scrub 2024-09-18T12:03:43.015 INFO:tasks.ceph:Scrubbing osd.5 2024-09-18T12:03:43.015 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:03:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 10 monclient: get_monmap_and_config 2024-09-18T12:03:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 10 monclient: build_initial_monmap 2024-09-18T12:03:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 10 monclient: monmap: 2024-09-18T12:03:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:43.130316+0000 2024-09-18T12:03:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:43.130316+0000 2024-09-18T12:03:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:43.130 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 1 Processor -- start 2024-09-18T12:03:43.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 1 -- start start 2024-09-18T12:03:43.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 10 monclient: init 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-b con 0x7fdbc40a3820 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.125+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-c con 0x7fdbc40a3bf0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-a con 0x7fdbc4138430 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40a53e0 con 0x7fdbc4138430 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc4096d60 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc4096a60 con 0x7fdbc40a3820 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 10 monclient(hunting): authenticate will time out at 1994.972092s 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc2d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdbc4138430 0x7fdbc413a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46222/0 (socket says 172.21.15.112:46222) 2024-09-18T12:03:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc2d9d700 1 -- 172.21.15.112:0/240215126 learned_addr learned my addr 172.21.15.112:0/240215126 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:43.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc2d9d700 1 -- 172.21.15.112:0/240215126 >> v1:172.21.15.112:6789/0 conn(0x7fdbc4138430 legacy=0x7fdbc413a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc259c700 1 -- 172.21.15.112:0/240215126 >> v1:172.21.15.112:6790/0 conn(0x7fdbc40a3820 legacy=0x7fdbc4097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201260705 0 0) 0x7fdbc40a53e0 con 0x7fdbc4138430 2024-09-18T12:03:43.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): my global_id is 164220 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc1d9b700 1 -- 172.21.15.112:0/240215126 >> v1:172.21.15.129:6789/0 conn(0x7fdbc40a3bf0 legacy=0x7fdbc40a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbac003570 con 0x7fdbc4138430 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592040346 0 0) 0x7fdbc4096a60 con 0x7fdbc40a3820 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): my global_id is 154316 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc40a53e0 con 0x7fdbc40a3820 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116855488 0 0) 0x7fdbc4096d60 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient(hunting): my global_id is 144403 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc4096a60 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 865638319 0 0) 0x7fdbc40a53e0 con 0x7fdbc40a3820 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 >> v1:172.21.15.129:6789/0 conn(0x7fdbc40a3bf0 legacy=0x7fdbc40a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 >> v1:172.21.15.112:6789/0 conn(0x7fdbc4138430 legacy=0x7fdbc413a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: _finish_hunting 0 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 monclient: found mon.noname-b 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40a4700 con 0x7fdbc40a3820 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: _finish_auth 0 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 5 monclient: authenticate success, global_id 154316 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdbb8001970 con 0x7fdbc40a3820 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: dump: 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbb8001c00 con 0x7fdbc40a3820 2024-09-18T12:03:43.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc359e700 4 set_mon_vals no callback set 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 1 -- 172.21.15.112:0/240215126 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdbb80020a0 con 0x7fdbc40a3820 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.129+0000 7fdbc159a700 10 monclient: dump: 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.140 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient: shutdown 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- 172.21.15.112:0/240215126 >> v1:172.21.15.112:6790/0 conn(0x7fdbc40a3820 legacy=0x7fdbc4097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- 172.21.15.112:0/240215126 shutdown_connections 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- 172.21.15.112:0/240215126 >> 172.21.15.112:0/240215126 conn(0x7fdbc40a5a20 msgr2=0x7fdbc40a5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- 172.21.15.112:0/240215126 shutdown_connections 2024-09-18T12:03:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- 172.21.15.112:0/240215126 wait complete. 2024-09-18T12:03:43.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient: build_initial_monmap 2024-09-18T12:03:43.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient: monmap: 2024-09-18T12:03:43.147 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:43.147 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:43.147 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:43.147 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 librados: starting msgr at 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 librados: starting objecter 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 Processor -- start 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- start start 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 librados: setting wanted keys 2024-09-18T12:03:43.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 librados: calling monclient init 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient: init 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-c con 0x7fdbc40a3820 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-b con 0x7fdbc40a3bf0 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-a con 0x7fdbc4138430 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc409ecc0 con 0x7fdbc4138430 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc4142440 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc4098260 con 0x7fdbc40a3820 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc359e700 10 monclient(hunting): authenticate will time out at 1994.979790s 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc259c700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fdbc40a3820 0x7fdbc4141540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58874/0 (socket says 172.21.15.112:58874) 2024-09-18T12:03:43.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.133+0000 7fdbc259c700 1 -- 172.21.15.112:0/356782636 learned_addr learned my addr 172.21.15.112:0/356782636 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc259c700 1 -- 172.21.15.112:0/356782636 >> v1:172.21.15.112:6790/0 conn(0x7fdbc40a3820 legacy=0x7fdbc4141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 client.?.objecter ms_handle_connect 0x7fdbc40a3820 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 client.?.objecter ms_handle_connect 0x7fdbc4138430 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc2d9d700 1 -- 172.21.15.112:0/356782636 >> v1:172.21.15.112:6789/0 conn(0x7fdbc4138430 legacy=0x7fdbc41498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc1d9b700 1 -- 172.21.15.112:0/356782636 >> v1:172.21.15.129:6789/0 conn(0x7fdbc40a3bf0 legacy=0x7fdbc409e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 client.?.objecter ms_handle_connect 0x7fdbc40a3bf0 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947910666 0 0) 0x7fdbc4098260 con 0x7fdbc40a3820 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): my global_id is 154319 2024-09-18T12:03:43.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba4004f10 con 0x7fdbc40a3820 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559334751 0 0) 0x7fdbc409ecc0 con 0x7fdbc4138430 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): my global_id is 164223 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc4098260 con 0x7fdbc4138430 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107826418 0 0) 0x7fdbc4142440 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): my global_id is 144406 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc409ecc0 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320781893 0 0) 0x7fdba4004f10 con 0x7fdbc40a3820 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc4142440 con 0x7fdbc40a3820 2024-09-18T12:03:43.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdbb80020a0 con 0x7fdbc40a3820 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): dump: 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.152 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1617856437 0 0) 0x7fdbc4098260 con 0x7fdbc4138430 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdba4004f10 con 0x7fdbc4138430 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdbbc004bb0 con 0x7fdbc4138430 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): dump: 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2952615953 0 0) 0x7fdbc409ecc0 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc4098260 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdbb4001bc0 con 0x7fdbc40a3bf0 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient(hunting): dump: 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1484650457 0 0) 0x7fdbc4142440 con 0x7fdbc40a3820 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 >> v1:172.21.15.129:6789/0 conn(0x7fdbc40a3bf0 legacy=0x7fdbc409e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 >> v1:172.21.15.112:6789/0 conn(0x7fdbc4138430 legacy=0x7fdbc41498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient: _finish_hunting 0 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 monclient: found mon.c 2024-09-18T12:03:43.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc4098540 con 0x7fdbc40a3820 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 10 monclient: _finish_auth 0 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 5 monclient: authenticate success, global_id 154319 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 10 monclient: _renew_subs 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc4137d20 con 0x7fdbc40a3820 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 10 client.154319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 10 monclient: _renew_subs 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc409e780 con 0x7fdbc40a3820 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbc359e700 1 librados: init done 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.137+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbb8002650 con 0x7fdbc40a3820 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fdbb8002a20 con 0x7fdbc40a3820 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc0d99700 4 set_mon_vals no callback set 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 10 monclient: dump: 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.157 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc359e700 10 client.154319.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc359e700 15 client.154319.objecter _session_command_op_assign -1 1 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc359e700 20 client.154319.objecter _get_session osd=-1 returning homeless 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc359e700 10 client.154319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.112:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdba80028c0 con 0x7fdbc40a3820 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fdbb8002c00 con 0x7fdbc40a3820 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fdbb8043e50 con 0x7fdbc40a3820 2024-09-18T12:03:43.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 10 client.154319.objecter ms_dispatch 0x7fdbc4000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 3 client.154319.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 3 client.154319.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 10 client.154319.objecter checking command 1 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 20 client.154319.objecter _get_session s=0x7fdba4040120 osd=5 3 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 20 client.154319.objecter put_session s=0x7fdba4040120 osd=5 3 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 15 client.154319.objecter _session_command_op_remove -1 1 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 20 client.154319.objecter _get_session s=0x7fdba4040120 osd=5 3 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 20 client.154319.objecter get_session s=0x7fdba4040120 osd=5 3 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 15 client.154319.objecter _session_command_op_assign 5 1 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 20 client.154319.objecter put_session s=0x7fdba4040120 osd=5 4 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 10 client.154319.objecter _send_command 1 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.129:6800/21048 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdba40444d0 con 0x7fdba40407d0 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 20 client.154319.objecter dump_active .. 0 homeless 2024-09-18T12:03:43.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_get_version_reply(handle=1 version=699) v2 ==== 24+0+0 (unknown 757610054 0 0) 0x7fdbb8044280 con 0x7fdbc40a3820 2024-09-18T12:03:43.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbb27fc700 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-18T12:03:43.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc1d9b700 10 monclient: get_auth_request con 0x7fdba40385e0 auth_method 0 2024-09-18T12:03:43.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.141+0000 7fdbc2d9d700 10 monclient: get_auth_request con 0x7fdba40407d0 auth_method 0 2024-09-18T12:03:43.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.145+0000 7fdbb27fc700 10 client.154319.objecter ms_handle_connect 0x7fdba40385e0 2024-09-18T12:03:43.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.145+0000 7fdbb27fc700 10 client.154319.objecter ms_handle_connect 0x7fdba40407d0 2024-09-18T12:03:43.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.149+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== osd.5 v1:172.21.15.129:6800/21048 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdba40444d0 con 0x7fdba40407d0 2024-09-18T12:03:43.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.149+0000 7fdbb27fc700 10 client.154319.objecter ms_dispatch 0x7fdbc4000fd0 command_reply(tid 1: 0 ) v1 2024-09-18T12:03:43.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.149+0000 7fdbb27fc700 10 client.154319.objecter _finish_command 1 = system:0 2024-09-18T12:03:43.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.149+0000 7fdbb27fc700 20 client.154319.objecter put_session s=0x7fdba4040120 osd=5 4 2024-09-18T12:03:43.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.149+0000 7fdbb27fc700 15 client.154319.objecter _session_command_op_remove 5 1 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 10 client.154319.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 15 client.154319.objecter _session_command_op_assign -1 2 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 20 client.154319.objecter _get_session s=0x7fdba4040120 osd=5 3 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 20 client.154319.objecter put_session s=0x7fdba4040120 osd=5 3 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 20 client.154319.objecter _get_session s=0x7fdba4040120 osd=5 3 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 15 client.154319.objecter _session_command_op_remove -1 2 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 20 client.154319.objecter get_session s=0x7fdba4040120 osd=5 3 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 15 client.154319.objecter _session_command_op_assign 5 2 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 20 client.154319.objecter put_session s=0x7fdba4040120 osd=5 4 2024-09-18T12:03:43.171 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 10 client.154319.objecter _send_command 2 2024-09-18T12:03:43.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.165+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 --> v1:172.21.15.129:6800/21048 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdba80011f0 con 0x7fdba40407d0 2024-09-18T12:03:43.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbb27fc700 1 -- 172.21.15.112:0/356782636 <== osd.5 v1:172.21.15.129:6800/21048 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7fdba80011f0 con 0x7fdba40407d0 2024-09-18T12:03:43.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbb27fc700 10 client.154319.objecter ms_dispatch 0x7fdbc4000fd0 command_reply(tid 2: 0 ) v1 2024-09-18T12:03:43.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbb27fc700 10 client.154319.objecter _finish_command 2 = system:0 2024-09-18T12:03:43.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbb27fc700 20 client.154319.objecter put_session s=0x7fdba4040120 osd=5 4 2024-09-18T12:03:43.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbb27fc700 15 client.154319.objecter _session_command_op_remove 5 2 2024-09-18T12:03:43.173 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-18T12:03:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:03:43.173 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-18T12:03:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 10 librados: watch_flush enter 2024-09-18T12:03:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 10 librados: watch_flush exit 2024-09-18T12:03:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 10 client.154319.objecter close_session for osd.5 2024-09-18T12:03:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 >> v1:172.21.15.129:6800/21048 conn(0x7fdba40407d0 legacy=0x7fdba4042c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 20 client.154319.objecter put_session s=0x7fdba4040120 osd=5 1 2024-09-18T12:03:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 20 client.154319.objecter shutdown clearing up homeless session... 2024-09-18T12:03:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 10 client.154319.objecter successfully canceled tick 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 >> v1:172.21.15.129:6812/20806 conn(0x7fdba40385e0 legacy=0x7fdba403aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 10 monclient: shutdown 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 >> v1:172.21.15.112:6790/0 conn(0x7fdbc40a3820 legacy=0x7fdbc4141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc2d9d700 1 -- 172.21.15.112:0/356782636 reap_dead start 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 shutdown_connections 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 >> 172.21.15.112:0/356782636 conn(0x7fdbc40a5a20 msgr2=0x7fdbc4138d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 shutdown_connections 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 1 -- 172.21.15.112:0/356782636 wait complete. 2024-09-18T12:03:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.169+0000 7fdbc359e700 1 librados: shutdown 2024-09-18T12:03:43.185 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.289+0000 7fe80caf7700 10 monclient: get_monmap_and_config 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.289+0000 7fe80caf7700 10 monclient: build_initial_monmap 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.289+0000 7fe80caf7700 10 monclient: monmap: 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:43.295980+0000 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:43.295980+0000 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:43.295 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:43.296 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:43.296 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 1 Processor -- start 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 1 -- start start 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 10 monclient: init 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 10 monclient(hunting): picked mon.noname-a con 0x7fe8080a0380 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 10 monclient(hunting): picked mon.noname-c con 0x7fe8080a0750 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 10 monclient(hunting): picked mon.noname-b con 0x7fe8081385b0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe808097270 con 0x7fe8080a0380 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8080a3170 con 0x7fe8080a0750 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe808096cb0 con 0x7fe8081385b0 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:43.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe80caf7700 10 monclient(hunting): authenticate will time out at 1995.137137s 2024-09-18T12:03:43.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe8077fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe8080a0380 0x7fe808097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46238/0 (socket says 172.21.15.112:46238) 2024-09-18T12:03:43.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe807fff700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fe8081385b0 0x7fe80813a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58880/0 (socket says 172.21.15.112:58880) 2024-09-18T12:03:43.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe8077fe700 1 -- 172.21.15.112:0/1658273329 learned_addr learned my addr 172.21.15.112:0/1658273329 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe807fff700 1 -- 172.21.15.112:0/1658273329 >> v1:172.21.15.112:6790/0 conn(0x7fe8081385b0 legacy=0x7fe80813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe8077fe700 1 -- 172.21.15.112:0/1658273329 >> v1:172.21.15.112:6789/0 conn(0x7fe8080a0380 legacy=0x7fe808097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe806ffd700 1 -- 172.21.15.112:0/1658273329 >> v1:172.21.15.129:6789/0 conn(0x7fe8080a0750 legacy=0x7fe8080a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248592031 0 0) 0x7fe808096cb0 con 0x7fe8081385b0 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe8067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.293+0000 7fe8067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient(hunting): my global_id is 154322 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f4003570 con 0x7fe8081385b0 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003048859 0 0) 0x7fe808097270 con 0x7fe8080a0380 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient(hunting): my global_id is 164226 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808096cb0 con 0x7fe8080a0380 2024-09-18T12:03:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389015969 0 0) 0x7fe8080a3170 con 0x7fe8080a0750 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient(hunting): my global_id is 144409 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808097270 con 0x7fe8080a0750 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3847625142 0 0) 0x7fe7f4003570 con 0x7fe8081385b0 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 >> v1:172.21.15.129:6789/0 conn(0x7fe8080a0750 legacy=0x7fe8080a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 >> v1:172.21.15.112:6789/0 conn(0x7fe8080a0380 legacy=0x7fe808097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: _finish_hunting 0 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 monclient: found mon.noname-b 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8080a1420 con 0x7fe8081385b0 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: _finish_auth 0 2024-09-18T12:03:43.303 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 5 monclient: authenticate success, global_id 154322 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7f0001940 con 0x7fe8081385b0 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: dump: 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.304 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7f0001bd0 con 0x7fe8081385b0 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 4 set_mon_vals no callback set 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 1 -- 172.21.15.112:0/1658273329 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7f00049d0 con 0x7fe8081385b0 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe8067fc700 10 monclient: dump: 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.305 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient: shutdown 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- 172.21.15.112:0/1658273329 >> v1:172.21.15.112:6790/0 conn(0x7fe8081385b0 legacy=0x7fe80813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- 172.21.15.112:0/1658273329 shutdown_connections 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- 172.21.15.112:0/1658273329 >> 172.21.15.112:0/1658273329 conn(0x7fe8080a5990 msgr2=0x7fe8080a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:43.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- 172.21.15.112:0/1658273329 shutdown_connections 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- 172.21.15.112:0/1658273329 wait complete. 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient: build_initial_monmap 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient: monmap: 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 librados: starting msgr at 2024-09-18T12:03:43.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 librados: starting objecter 2024-09-18T12:03:43.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 Processor -- start 2024-09-18T12:03:43.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- start start 2024-09-18T12:03:43.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 librados: setting wanted keys 2024-09-18T12:03:43.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 librados: calling monclient init 2024-09-18T12:03:43.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient: init 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient(hunting): picked mon.noname-b con 0x7fe8080a0380 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient(hunting): picked mon.noname-a con 0x7fe8080a0750 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient(hunting): picked mon.noname-c con 0x7fe8081385b0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8080a2010 con 0x7fe8080a0750 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe808068380 con 0x7fe8080a0380 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80811eb70 con 0x7fe8081385b0 2024-09-18T12:03:43.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe80caf7700 10 monclient(hunting): authenticate will time out at 1995.143532s 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe806ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe8080a0750 0x7fe808146800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46244/0 (socket says 172.21.15.112:46244) 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.297+0000 7fe806ffd700 1 -- 172.21.15.112:0/3248806478 learned_addr learned my addr 172.21.15.112:0/3248806478 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe806ffd700 1 -- 172.21.15.112:0/3248806478 >> v1:172.21.15.112:6789/0 conn(0x7fe8080a0750 legacy=0x7fe808146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 client.?.objecter ms_handle_connect 0x7fe8080a0750 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 client.?.objecter ms_handle_connect 0x7fe8081385b0 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe807fff700 1 -- 172.21.15.112:0/3248806478 >> v1:172.21.15.112:6790/0 conn(0x7fe8081385b0 legacy=0x7fe808149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165847899 0 0) 0x7fe80811eb70 con 0x7fe8081385b0 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): my global_id is 154325 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe8077fe700 1 -- 172.21.15.112:0/3248806478 >> v1:172.21.15.129:6789/0 conn(0x7fe8080a0380 legacy=0x7fe808141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e8004f10 con 0x7fe8081385b0 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 client.?.objecter ms_handle_connect 0x7fe8080a0380 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606041982 0 0) 0x7fe8080a2010 con 0x7fe8080a0750 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): my global_id is 164229 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe80811eb70 con 0x7fe8080a0750 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850798928 0 0) 0x7fe808068380 con 0x7fe8080a0380 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): my global_id is 144412 2024-09-18T12:03:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8080a2010 con 0x7fe8080a0380 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 896725045 0 0) 0x7fe7e8004f10 con 0x7fe8081385b0 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe808068380 con 0x7fe8081385b0 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7f0004830 con 0x7fe8081385b0 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): dump: 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2108321142 0 0) 0x7fe808068380 con 0x7fe8081385b0 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 >> v1:172.21.15.129:6789/0 conn(0x7fe8080a0380 legacy=0x7fe808141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 >> v1:172.21.15.112:6789/0 conn(0x7fe8080a0750 legacy=0x7fe808146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 monclient: found mon.c 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80811ee50 con 0x7fe8081385b0 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient: _finish_auth 0 2024-09-18T12:03:43.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 5 monclient: authenticate success, global_id 154325 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 10 monclient: _renew_subs 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80809e0c0 con 0x7fe8081385b0 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 10 client.154325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 10 monclient: _renew_subs 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80811ddb0 con 0x7fe8081385b0 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe80caf7700 1 librados: init done 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7f0004e80 con 0x7fe8081385b0 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fe7f0005190 con 0x7fe8081385b0 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe805ffb700 4 set_mon_vals no callback set 2024-09-18T12:03:43.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.301+0000 7fe7ff7fe700 10 monclient: dump: 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe80caf7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe80caf7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:43.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fe7f000fa10 con 0x7fe8081385b0 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe80caf7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec0050b0 con 0x7fe8081385b0 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fe7f0043b90 con 0x7fe8081385b0 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe7ff7fe700 10 client.154325.objecter ms_dispatch 0x7fe808000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe7ff7fe700 3 client.154325.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe7ff7fe700 3 client.154325.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe7ff7fe700 20 client.154325.objecter dump_active .. 0 homeless 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe8077fe700 10 monclient: get_auth_request con 0x7fe7e8038100 auth_method 0 2024-09-18T12:03:43.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.305+0000 7fe7ff7fe700 10 client.154325.objecter ms_handle_connect 0x7fe7e8038100 2024-09-18T12:03:43.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.313+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fe7f0014b10 con 0x7fe8081385b0 2024-09-18T12:03:43.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.313+0000 7fe7ff7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:43.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.313+0000 7fe7ff7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:43.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.477+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7ec000b90 con 0x7fe7e8038100 2024-09-18T12:03:43.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.477+0000 7fe7ff7fe700 1 -- 172.21.15.112:0/3248806478 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (unknown 2264484800 0 0) 0x7fe7ec000b90 con 0x7fe7e8038100 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 10 librados: watch_flush enter 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 10 librados: watch_flush exit 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 20 client.154325.objecter shutdown clearing up homeless session... 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 10 client.154325.objecter successfully canceled tick 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 >> v1:172.21.15.129:6812/20806 conn(0x7fe7e8038100 legacy=0x7fe7e803a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 10 monclient: shutdown 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 >> v1:172.21.15.112:6790/0 conn(0x7fe8081385b0 legacy=0x7fe808149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 shutdown_connections 2024-09-18T12:03:43.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 >> 172.21.15.112:0/3248806478 conn(0x7fe8080a5990 msgr2=0x7fe80809ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:43.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 shutdown_connections 2024-09-18T12:03:43.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 1 -- 172.21.15.112:0/3248806478 wait complete. 2024-09-18T12:03:43.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.481+0000 7fe80caf7700 1 librados: shutdown 2024-09-18T12:03:43.486 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 5 to deep-scrub 2024-09-18T12:03:43.501 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:43.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient: get_monmap_and_config 2024-09-18T12:03:43.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient: build_initial_monmap 2024-09-18T12:03:43.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient: monmap: 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:03:43.608652+0000 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:03:43.608652+0000 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:03:43.608 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 1 Processor -- start 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 1 -- start start 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient: init 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient(hunting): picked mon.noname-c con 0x7ff8ac06aa40 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient(hunting): picked mon.noname-b con 0x7ff8ac0a1e80 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient(hunting): picked mon.noname-a con 0x7ff8ac1386e0 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac097270 con 0x7ff8ac1386e0 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac06ae10 con 0x7ff8ac06aa40 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac097550 con 0x7ff8ac0a1e80 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:43.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b47be700 10 monclient(hunting): authenticate will time out at 1995.449700s 2024-09-18T12:03:43.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b1d59700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7ff8ac0a1e80 0x7ff8ac0a1160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:58898/0 (socket says 172.21.15.112:58898) 2024-09-18T12:03:43.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.605+0000 7ff8b1d59700 1 -- 172.21.15.112:0/2756483676 learned_addr learned my addr 172.21.15.112:0/2756483676 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:43.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b2d5b700 1 -- 172.21.15.112:0/2756483676 >> v1:172.21.15.112:6789/0 conn(0x7ff8ac1386e0 legacy=0x7ff8ac13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1d59700 1 -- 172.21.15.112:0/2756483676 >> v1:172.21.15.112:6790/0 conn(0x7ff8ac0a1e80 legacy=0x7ff8ac0a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b255a700 1 -- 172.21.15.112:0/2756483676 >> v1:172.21.15.129:6789/0 conn(0x7ff8ac06aa40 legacy=0x7ff8ac0a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1d59700 1 -- 172.21.15.112:0/2756483676 >> v1:172.21.15.112:6790/0 conn(0x7ff8ac0a1e80 legacy=0x7ff8ac0a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1d59700 1 -- 172.21.15.112:0/2756483676 >> v1:172.21.15.112:6790/0 conn(0x7ff8ac0a1e80 legacy=0x7ff8ac0a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1d59700 1 --1- 172.21.15.112:0/2756483676 >> v1:172.21.15.112:6790/0 conn(0x7ff8ac0a1e80 0x7ff8ac0a1160 :-1 s=OPENED pgs=9379 cs=1 l=1).handle_message read tag failed 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1d59700 1 --1- 172.21.15.112:0/2756483676 >> v1:172.21.15.112:6790/0 conn(0x7ff8ac0a1e80 0x7ff8ac0a1160 :-1 s=OPENED pgs=9379 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670833955 0 0) 0x7ff8ac097270 con 0x7ff8ac1386e0 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient(hunting): my global_id is 164232 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff894003570 con 0x7ff8ac1386e0 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6790/0 2024-09-18T12:03:43.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919962234 0 0) 0x7ff8ac06ae10 con 0x7ff8ac06aa40 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient(hunting): my global_id is 144415 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac097270 con 0x7ff8ac06aa40 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4243944348 0 0) 0x7ff8ac097270 con 0x7ff8ac06aa40 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 >> v1:172.21.15.112:6790/0 conn(0x7ff8ac0a1e80 legacy=0x7ff8ac0a1160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 >> v1:172.21.15.112:6789/0 conn(0x7ff8ac1386e0 legacy=0x7ff8ac13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: _finish_hunting 0 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 monclient: found mon.noname-c 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac096a60 con 0x7ff8ac06aa40 2024-09-18T12:03:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: _finish_auth 0 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 5 monclient: authenticate success, global_id 144415 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff89c0019f0 con 0x7ff8ac06aa40 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: dump: 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.616 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff89c001b50 con 0x7ff8ac06aa40 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 1 -- 172.21.15.112:0/2756483676 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff89c001f00 con 0x7ff8ac06aa40 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b1558700 10 monclient: dump: 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.617 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 10 monclient: get_monmap_and_config success 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 4 set_mon_vals no callback set 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 10 monclient: shutdown 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 1 -- 172.21.15.112:0/2756483676 >> v1:172.21.15.129:6789/0 conn(0x7ff8ac06aa40 legacy=0x7ff8ac0a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 1 -- 172.21.15.112:0/2756483676 shutdown_connections 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 1 -- 172.21.15.112:0/2756483676 >> 172.21.15.112:0/2756483676 conn(0x7ff8ac0a5d10 msgr2=0x7ff8ac0a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:43.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 1 -- 172.21.15.112:0/2756483676 shutdown_connections 2024-09-18T12:03:43.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 1 -- 172.21.15.112:0/2756483676 wait complete. 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 10 monclient: build_initial_monmap 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 10 monclient: monmap: 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:03:43.623 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 1 librados: starting msgr at 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 1 librados: starting objecter 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.609+0000 7ff8b47be700 1 Processor -- start 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 1 -- start start 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 1 librados: setting wanted keys 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 1 librados: calling monclient init 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 10 monclient: init 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 10 monclient: _reopen_session rank -1 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:03:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 10 monclient(hunting): picked mon.noname-a con 0x7ff8ac06aa40 addr v1:172.21.15.112:6789/0 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 10 monclient(hunting): picked mon.noname-b con 0x7ff8ac0a1e80 addr v1:172.21.15.129:6789/0 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 10 monclient(hunting): picked mon.noname-c con 0x7ff8ac1386e0 addr v1:172.21.15.112:6790/0 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac09b630 con 0x7ff8ac06aa40 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b255a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff8ac06aa40 0x7ff8ac09e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:46262/0 (socket says 172.21.15.112:46262) 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b255a700 1 -- 172.21.15.112:0/1755935183 learned_addr learned my addr 172.21.15.112:0/1755935183 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac09a020 con 0x7ff8ac0a1e80 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0a22f0 con 0x7ff8ac1386e0 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 10 monclient(hunting): _renew_subs 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b47be700 10 monclient(hunting): authenticate will time out at 1995.457545s 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b255a700 1 -- 172.21.15.112:0/1755935183 >> v1:172.21.15.112:6789/0 conn(0x7ff8ac06aa40 legacy=0x7ff8ac09e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 client.?.objecter ms_handle_connect 0x7ff8ac06aa40 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b2d5b700 1 -- 172.21.15.112:0/1755935183 >> v1:172.21.15.112:6790/0 conn(0x7ff8ac1386e0 legacy=0x7ff8ac14a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 client.?.objecter ms_handle_connect 0x7ff8ac1386e0 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8b1d59700 1 -- 172.21.15.112:0/1755935183 >> v1:172.21.15.129:6789/0 conn(0x7ff8ac0a1e80 legacy=0x7ff8ac146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210103431 0 0) 0x7ff8ac09b630 con 0x7ff8ac06aa40 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 monclient(hunting): my global_id is 164235 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff898004f10 con 0x7ff8ac06aa40 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 client.?.objecter ms_handle_connect 0x7ff8ac0a1e80 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.613+0000 7ff8aa7fc700 10 client.?.objecter resend_mon_ops 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839616602 0 0) 0x7ff8ac0a22f0 con 0x7ff8ac1386e0 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.626 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): my global_id is 154331 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac09b630 con 0x7ff8ac1386e0 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547403269 0 0) 0x7ff8ac09a020 con 0x7ff8ac0a1e80 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): my global_id is 144418 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac0a22f0 con 0x7ff8ac0a1e80 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 90249012 0 0) 0x7ff898004f10 con 0x7ff8ac06aa40 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ac09a020 con 0x7ff8ac06aa40 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff89c002270 con 0x7ff8ac06aa40 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): dump: 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.627 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2390545475 0 0) 0x7ff8ac09b630 con 0x7ff8ac1386e0 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff898004f10 con 0x7ff8ac1386e0 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff8a4001cc0 con 0x7ff8ac1386e0 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.628 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): dump: 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 793297936 0 0) 0x7ff8ac0a22f0 con 0x7ff8ac0a1e80 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ac09b630 con 0x7ff8ac0a1e80 2024-09-18T12:03:43.629 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff8a0001be0 con 0x7ff8ac0a1e80 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient(hunting): dump: 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 647909710 0 0) 0x7ff8ac09a020 con 0x7ff8ac06aa40 2024-09-18T12:03:43.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 >> v1:172.21.15.112:6790/0 conn(0x7ff8ac1386e0 legacy=0x7ff8ac14a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 >> v1:172.21.15.129:6789/0 conn(0x7ff8ac0a1e80 legacy=0x7ff8ac146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient: _finish_hunting 0 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 monclient: found mon.a 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac09b350 con 0x7ff8ac06aa40 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient: _finish_auth 0 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 5 monclient: authenticate success, global_id 164235 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 10 monclient: _renew_subs 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac09a4c0 con 0x7ff8ac06aa40 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 10 client.164235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:03:43.631 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 10 monclient: _renew_subs 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac142490 con 0x7ff8ac06aa40 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 1 librados: init done 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff89c002680 con 0x7ff8ac06aa40 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7ff89c002ad0 con 0x7ff8ac06aa40 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b0d57700 4 set_mon_vals no callback set 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 monclient: dump: 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.632 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7ff89c00ff30 con 0x7ff8ac06aa40 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c0050b0 con 0x7ff8ac06aa40 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7ff89c043df0 con 0x7ff8ac06aa40 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 10 client.164235.objecter ms_dispatch 0x7ff8ac000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:03:43.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 3 client.164235.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:03:43.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 3 client.164235.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:03:43.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8aa7fc700 20 client.164235.objecter dump_active .. 0 homeless 2024-09-18T12:03:43.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.617+0000 7ff8b1d59700 10 monclient: get_auth_request con 0x7ff8980385b0 auth_method 0 2024-09-18T12:03:43.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.621+0000 7ff8aa7fc700 10 client.164235.objecter ms_handle_connect 0x7ff8980385b0 2024-09-18T12:03:43.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.629+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7ff89c002cb0 con 0x7ff8ac06aa40 2024-09-18T12:03:43.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.629+0000 7ff8aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:03:43.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.629+0000 7ff8aa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:03:43.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.789+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff88c000b90 con 0x7ff8980385b0 2024-09-18T12:03:43.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.797+0000 7ff8aa7fc700 1 -- 172.21.15.112:0/1755935183 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134063 (unknown 2965378022 0 2909089736) 0x7ff88c000b90 con 0x7ff8980385b0 2024-09-18T12:03:43.804 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T12:03:43.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 10 librados: watch_flush enter 2024-09-18T12:03:43.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 10 librados: watch_flush exit 2024-09-18T12:03:43.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 20 client.164235.objecter shutdown clearing up homeless session... 2024-09-18T12:03:43.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 10 client.164235.objecter successfully canceled tick 2024-09-18T12:03:43.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 >> v1:172.21.15.129:6812/20806 conn(0x7ff8980385b0 legacy=0x7ff89803aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 10 monclient: shutdown 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 >> v1:172.21.15.112:6789/0 conn(0x7ff8ac06aa40 legacy=0x7ff8ac09e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 shutdown_connections 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 >> 172.21.15.112:0/1755935183 conn(0x7ff8ac0a5d10 msgr2=0x7ff8ac13aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 shutdown_connections 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 1 -- 172.21.15.112:0/1755935183 wait complete. 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:03:43.801+0000 7ff8b47be700 1 librados: shutdown 2024-09-18T12:03:43.806 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T12:03:43.818 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1276,"stamp":"2024-09-18T12:03:42.382187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1898908,"kb_used_data":56444,"kb_used_omap":9,"kb_used_meta":1842294,"kb_avail":564332132,"statfs":{"total":579820584960,"available":577876103168,"internally_reserved":0,"allocated":57798656,"data_stored":30786034,"data_compressed":320902,"data_compressed_allocated":876544,"data_compressed_original":2744320,"omap_allocated":10224,"internal_metadata":1886509072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003804"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522453+0000","last_change":"2024-09-18T11:45:25.374991+0000","last_active":"2024-09-18T12:02:11.522453+0000","last_peered":"2024-09-18T12:02:11.522453+0000","last_clean":"2024-09-18T12:02:11.522453+0000","last_became_active":"2024-09-18T11:45:25.373642+0000","last_became_peered":"2024-09-18T11:45:25.373642+0000","last_unstale":"2024-09-18T12:02:11.522453+0000","last_undegraded":"2024-09-18T12:02:11.522453+0000","last_fullsized":"2024-09-18T12:02:11.522453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:57:36.040393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519515+0000","last_change":"2024-09-18T11:45:25.329845+0000","last_active":"2024-09-18T12:02:11.519515+0000","last_peered":"2024-09-18T12:02:11.519515+0000","last_clean":"2024-09-18T12:02:11.519515+0000","last_became_active":"2024-09-18T11:45:25.328032+0000","last_became_peered":"2024-09-18T11:45:25.328032+0000","last_unstale":"2024-09-18T12:02:11.519515+0000","last_undegraded":"2024-09-18T12:02:11.519515+0000","last_fullsized":"2024-09-18T12:02:11.519515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:13.680321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913257+0000","last_change":"2024-09-18T11:45:25.306434+0000","last_active":"2024-09-18T12:02:10.913257+0000","last_peered":"2024-09-18T12:02:10.913257+0000","last_clean":"2024-09-18T12:02:10.913257+0000","last_became_active":"2024-09-18T11:45:25.305872+0000","last_became_peered":"2024-09-18T11:45:25.305872+0000","last_unstale":"2024-09-18T12:02:10.913257+0000","last_undegraded":"2024-09-18T12:02:10.913257+0000","last_fullsized":"2024-09-18T12:02:10.913257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:30:00.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519584+0000","last_change":"2024-09-18T11:45:25.411469+0000","last_active":"2024-09-18T12:02:11.519584+0000","last_peered":"2024-09-18T12:02:11.519584+0000","last_clean":"2024-09-18T12:02:11.519584+0000","last_became_active":"2024-09-18T11:45:25.411133+0000","last_became_peered":"2024-09-18T11:45:25.411133+0000","last_unstale":"2024-09-18T12:02:11.519584+0000","last_undegraded":"2024-09-18T12:02:11.519584+0000","last_fullsized":"2024-09-18T12:02:11.519584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:02:10.494289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523123+0000","last_change":"2024-09-18T11:45:25.400321+0000","last_active":"2024-09-18T12:02:11.523123+0000","last_peered":"2024-09-18T12:02:11.523123+0000","last_clean":"2024-09-18T12:02:11.523123+0000","last_became_active":"2024-09-18T11:45:25.399898+0000","last_became_peered":"2024-09-18T11:45:25.399898+0000","last_unstale":"2024-09-18T12:02:11.523123+0000","last_undegraded":"2024-09-18T12:02:11.523123+0000","last_fullsized":"2024-09-18T12:02:11.523123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:14.033638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.520050+0000","last_change":"2024-09-18T11:45:25.397202+0000","last_active":"2024-09-18T12:02:11.520050+0000","last_peered":"2024-09-18T12:02:11.520050+0000","last_clean":"2024-09-18T12:02:11.520050+0000","last_became_active":"2024-09-18T11:45:25.396975+0000","last_became_peered":"2024-09-18T11:45:25.396975+0000","last_unstale":"2024-09-18T12:02:11.520050+0000","last_undegraded":"2024-09-18T12:02:11.520050+0000","last_fullsized":"2024-09-18T12:02:11.520050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:37:37.634734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.910998+0000","last_change":"2024-09-18T11:45:25.341093+0000","last_active":"2024-09-18T12:02:10.910998+0000","last_peered":"2024-09-18T12:02:10.910998+0000","last_clean":"2024-09-18T12:02:10.910998+0000","last_became_active":"2024-09-18T11:45:25.340726+0000","last_became_peered":"2024-09-18T11:45:25.340726+0000","last_unstale":"2024-09-18T12:02:10.910998+0000","last_undegraded":"2024-09-18T12:02:10.910998+0000","last_fullsized":"2024-09-18T12:02:10.910998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:36:59.172751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960537+0000","last_change":"2024-09-18T11:45:25.408737+0000","last_active":"2024-09-18T12:02:10.960537+0000","last_peered":"2024-09-18T12:02:10.960537+0000","last_clean":"2024-09-18T12:02:10.960537+0000","last_became_active":"2024-09-18T11:45:25.408464+0000","last_became_peered":"2024-09-18T11:45:25.408464+0000","last_unstale":"2024-09-18T12:02:10.960537+0000","last_undegraded":"2024-09-18T12:02:10.960537+0000","last_fullsized":"2024-09-18T12:02:10.960537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:44:57.076431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.909867+0000","last_change":"2024-09-18T11:45:25.344531+0000","last_active":"2024-09-18T12:02:10.909867+0000","last_peered":"2024-09-18T12:02:10.909867+0000","last_clean":"2024-09-18T12:02:10.909867+0000","last_became_active":"2024-09-18T11:45:25.343097+0000","last_became_peered":"2024-09-18T11:45:25.343097+0000","last_unstale":"2024-09-18T12:02:10.909867+0000","last_undegraded":"2024-09-18T12:02:10.909867+0000","last_fullsized":"2024-09-18T12:02:10.909867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:49.438992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519649+0000","last_change":"2024-09-18T11:45:25.377493+0000","last_active":"2024-09-18T12:02:11.519649+0000","last_peered":"2024-09-18T12:02:11.519649+0000","last_clean":"2024-09-18T12:02:11.519649+0000","last_became_active":"2024-09-18T11:45:25.377194+0000","last_became_peered":"2024-09-18T11:45:25.377194+0000","last_unstale":"2024-09-18T12:02:11.519649+0000","last_undegraded":"2024-09-18T12:02:11.519649+0000","last_fullsized":"2024-09-18T12:02:11.519649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:44:53.107224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522770+0000","last_change":"2024-09-18T11:45:25.346404+0000","last_active":"2024-09-18T12:02:11.522770+0000","last_peered":"2024-09-18T12:02:11.522770+0000","last_clean":"2024-09-18T12:02:11.522770+0000","last_became_active":"2024-09-18T11:45:25.341682+0000","last_became_peered":"2024-09-18T11:45:25.341682+0000","last_unstale":"2024-09-18T12:02:11.522770+0000","last_undegraded":"2024-09-18T12:02:11.522770+0000","last_fullsized":"2024-09-18T12:02:11.522770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:04.010096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522588+0000","last_change":"2024-09-18T11:45:25.383080+0000","last_active":"2024-09-18T12:02:11.522588+0000","last_peered":"2024-09-18T12:02:11.522588+0000","last_clean":"2024-09-18T12:02:11.522588+0000","last_became_active":"2024-09-18T11:45:25.377663+0000","last_became_peered":"2024-09-18T11:45:25.377663+0000","last_unstale":"2024-09-18T12:02:11.522588+0000","last_undegraded":"2024-09-18T12:02:11.522588+0000","last_fullsized":"2024-09-18T12:02:11.522588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:46:12.443901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":649,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.914509+0000","last_change":"2024-09-18T11:44:24.299141+0000","last_active":"2024-09-18T12:02:10.914509+0000","last_peered":"2024-09-18T12:02:10.914509+0000","last_clean":"2024-09-18T12:02:10.914509+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T12:02:10.914509+0000","last_undegraded":"2024-09-18T12:02:10.914509+0000","last_fullsized":"2024-09-18T12:02:10.914509+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:35:52.704661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960533+0000","last_change":"2024-09-18T11:45:25.405730+0000","last_active":"2024-09-18T12:02:10.960533+0000","last_peered":"2024-09-18T12:02:10.960533+0000","last_clean":"2024-09-18T12:02:10.960533+0000","last_became_active":"2024-09-18T11:45:25.405515+0000","last_became_peered":"2024-09-18T11:45:25.405515+0000","last_unstale":"2024-09-18T12:02:10.960533+0000","last_undegraded":"2024-09-18T12:02:10.960533+0000","last_fullsized":"2024-09-18T12:02:10.960533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:39:09.362580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911684+0000","last_change":"2024-09-18T11:44:24.208317+0000","last_active":"2024-09-18T12:02:10.911684+0000","last_peered":"2024-09-18T12:02:10.911684+0000","last_clean":"2024-09-18T12:02:10.911684+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T12:02:10.911684+0000","last_undegraded":"2024-09-18T12:02:10.911684+0000","last_fullsized":"2024-09-18T12:02:10.911684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:54.979951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913034+0000","last_change":"2024-09-18T11:45:25.306008+0000","last_active":"2024-09-18T12:02:10.913034+0000","last_peered":"2024-09-18T12:02:10.913034+0000","last_clean":"2024-09-18T12:02:10.913034+0000","last_became_active":"2024-09-18T11:45:25.305686+0000","last_became_peered":"2024-09-18T11:45:25.305686+0000","last_unstale":"2024-09-18T12:02:10.913034+0000","last_undegraded":"2024-09-18T12:02:10.913034+0000","last_fullsized":"2024-09-18T12:02:10.913034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:31:41.349569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519991+0000","last_change":"2024-09-18T11:44:24.213224+0000","last_active":"2024-09-18T12:02:11.519991+0000","last_peered":"2024-09-18T12:02:11.519991+0000","last_clean":"2024-09-18T12:02:11.519991+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T12:02:11.519991+0000","last_undegraded":"2024-09-18T12:02:11.519991+0000","last_fullsized":"2024-09-18T12:02:11.519991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:47.310939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"20'7","reported_seq":660,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.910044+0000","last_change":"2024-09-18T11:45:25.403798+0000","last_active":"2024-09-18T12:02:10.910044+0000","last_peered":"2024-09-18T12:02:10.910044+0000","last_clean":"2024-09-18T12:02:10.910044+0000","last_became_active":"2024-09-18T11:45:25.403586+0000","last_became_peered":"2024-09-18T11:45:25.403586+0000","last_unstale":"2024-09-18T12:02:10.910044+0000","last_undegraded":"2024-09-18T12:02:10.910044+0000","last_fullsized":"2024-09-18T12:02:10.910044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:53:41.121574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":654,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522879+0000","last_change":"2024-09-18T11:44:24.507933+0000","last_active":"2024-09-18T12:02:11.522879+0000","last_peered":"2024-09-18T12:02:11.522879+0000","last_clean":"2024-09-18T12:02:11.522879+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T12:02:11.522879+0000","last_undegraded":"2024-09-18T12:02:11.522879+0000","last_fullsized":"2024-09-18T12:02:11.522879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:56.697013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522889+0000","last_change":"2024-09-18T11:45:25.369556+0000","last_active":"2024-09-18T12:02:11.522889+0000","last_peered":"2024-09-18T12:02:11.522889+0000","last_clean":"2024-09-18T12:02:11.522889+0000","last_became_active":"2024-09-18T11:45:25.367362+0000","last_became_peered":"2024-09-18T11:45:25.367362+0000","last_unstale":"2024-09-18T12:02:11.522889+0000","last_undegraded":"2024-09-18T12:02:11.522889+0000","last_fullsized":"2024-09-18T12:02:11.522889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:23:17.811113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":743,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.907687+0000","last_change":"2024-09-18T11:44:21.097910+0000","last_active":"2024-09-18T12:02:10.907687+0000","last_peered":"2024-09-18T12:02:10.907687+0000","last_clean":"2024-09-18T12:02:10.907687+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T12:02:10.907687+0000","last_undegraded":"2024-09-18T12:02:10.907687+0000","last_fullsized":"2024-09-18T12:02:10.907687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:48.394073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":655,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522634+0000","last_change":"2024-09-18T11:44:24.508079+0000","last_active":"2024-09-18T12:02:11.522634+0000","last_peered":"2024-09-18T12:02:11.522634+0000","last_clean":"2024-09-18T12:02:11.522634+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T12:02:11.522634+0000","last_undegraded":"2024-09-18T12:02:11.522634+0000","last_fullsized":"2024-09-18T12:02:11.522634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:41:30.648862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913627+0000","last_change":"2024-09-18T11:45:25.407247+0000","last_active":"2024-09-18T12:02:10.913627+0000","last_peered":"2024-09-18T12:02:10.913627+0000","last_clean":"2024-09-18T12:02:10.913627+0000","last_became_active":"2024-09-18T11:45:25.406430+0000","last_became_peered":"2024-09-18T11:45:25.406430+0000","last_unstale":"2024-09-18T12:02:10.913627+0000","last_undegraded":"2024-09-18T12:02:10.913627+0000","last_fullsized":"2024-09-18T12:02:10.913627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:56:03.017626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"17'2","reported_seq":660,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.519718+0000","last_change":"2024-09-18T11:44:24.214482+0000","last_active":"2024-09-18T12:02:11.519718+0000","last_peered":"2024-09-18T12:02:11.519718+0000","last_clean":"2024-09-18T12:02:11.519718+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T12:02:11.519718+0000","last_undegraded":"2024-09-18T12:02:11.519718+0000","last_fullsized":"2024-09-18T12:02:11.519718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:26:34.055867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523238+0000","last_change":"2024-09-18T11:45:25.365786+0000","last_active":"2024-09-18T12:02:11.523238+0000","last_peered":"2024-09-18T12:02:11.523238+0000","last_clean":"2024-09-18T12:02:11.523238+0000","last_became_active":"2024-09-18T11:45:25.363235+0000","last_became_peered":"2024-09-18T11:45:25.363235+0000","last_unstale":"2024-09-18T12:02:11.523238+0000","last_undegraded":"2024-09-18T12:02:11.523238+0000","last_fullsized":"2024-09-18T12:02:11.523238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:15:20.066742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":649,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.912589+0000","last_change":"2024-09-18T11:44:24.298975+0000","last_active":"2024-09-18T12:02:10.912589+0000","last_peered":"2024-09-18T12:02:10.912589+0000","last_clean":"2024-09-18T12:02:10.912589+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T12:02:10.912589+0000","last_undegraded":"2024-09-18T12:02:10.912589+0000","last_fullsized":"2024-09-18T12:02:10.912589+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:48:27.175495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960440+0000","last_change":"2024-09-18T11:45:25.356461+0000","last_active":"2024-09-18T12:02:10.960440+0000","last_peered":"2024-09-18T12:02:10.960440+0000","last_clean":"2024-09-18T12:02:10.960440+0000","last_became_active":"2024-09-18T11:45:25.356178+0000","last_became_peered":"2024-09-18T11:45:25.356178+0000","last_unstale":"2024-09-18T12:02:10.960440+0000","last_undegraded":"2024-09-18T12:02:10.960440+0000","last_fullsized":"2024-09-18T12:02:10.960440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:06:26.698280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":658,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.908011+0000","last_change":"2024-09-18T11:44:24.208252+0000","last_active":"2024-09-18T12:02:10.908011+0000","last_peered":"2024-09-18T12:02:10.908011+0000","last_clean":"2024-09-18T12:02:10.908011+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T12:02:10.908011+0000","last_undegraded":"2024-09-18T12:02:10.908011+0000","last_fullsized":"2024-09-18T12:02:10.908011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:44:20.059736+0000","last_clean_scrub_stamp":"2024-09-18T11:44:20.059736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:08:27.257954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.520056+0000","last_change":"2024-09-18T11:45:25.396966+0000","last_active":"2024-09-18T12:02:11.520056+0000","last_peered":"2024-09-18T12:02:11.520056+0000","last_clean":"2024-09-18T12:02:11.520056+0000","last_became_active":"2024-09-18T11:45:25.396428+0000","last_became_peered":"2024-09-18T11:45:25.396428+0000","last_unstale":"2024-09-18T12:02:11.520056+0000","last_undegraded":"2024-09-18T12:02:11.520056+0000","last_fullsized":"2024-09-18T12:02:11.520056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:08:22.504624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911592+0000","last_change":"2024-09-18T11:45:25.399933+0000","last_active":"2024-09-18T12:02:10.911592+0000","last_peered":"2024-09-18T12:02:10.911592+0000","last_clean":"2024-09-18T12:02:10.911592+0000","last_became_active":"2024-09-18T11:45:25.399695+0000","last_became_peered":"2024-09-18T11:45:25.399695+0000","last_unstale":"2024-09-18T12:02:10.911592+0000","last_undegraded":"2024-09-18T12:02:10.911592+0000","last_fullsized":"2024-09-18T12:02:10.911592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:01:15.698698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.913437+0000","last_change":"2024-09-18T11:45:25.387334+0000","last_active":"2024-09-18T12:02:10.913437+0000","last_peered":"2024-09-18T12:02:10.913437+0000","last_clean":"2024-09-18T12:02:10.913437+0000","last_became_active":"2024-09-18T11:45:25.386932+0000","last_became_peered":"2024-09-18T11:45:25.386932+0000","last_unstale":"2024-09-18T12:02:10.913437+0000","last_undegraded":"2024-09-18T12:02:10.913437+0000","last_fullsized":"2024-09-18T12:02:10.913437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:34:23.587779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":643,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522768+0000","last_change":"2024-09-18T11:45:25.403356+0000","last_active":"2024-09-18T12:02:11.522768+0000","last_peered":"2024-09-18T12:02:11.522768+0000","last_clean":"2024-09-18T12:02:11.522768+0000","last_became_active":"2024-09-18T11:45:25.401979+0000","last_became_peered":"2024-09-18T11:45:25.401979+0000","last_unstale":"2024-09-18T12:02:11.522768+0000","last_undegraded":"2024-09-18T12:02:11.522768+0000","last_fullsized":"2024-09-18T12:02:11.522768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:29:14.433201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523103+0000","last_change":"2024-09-18T11:45:25.381976+0000","last_active":"2024-09-18T12:02:11.523103+0000","last_peered":"2024-09-18T12:02:11.523103+0000","last_clean":"2024-09-18T12:02:11.523103+0000","last_became_active":"2024-09-18T11:45:25.381652+0000","last_became_peered":"2024-09-18T11:45:25.381652+0000","last_unstale":"2024-09-18T12:02:11.523103+0000","last_undegraded":"2024-09-18T12:02:11.523103+0000","last_fullsized":"2024-09-18T12:02:11.523103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:21.420291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522937+0000","last_change":"2024-09-18T11:45:25.381748+0000","last_active":"2024-09-18T12:02:11.522937+0000","last_peered":"2024-09-18T12:02:11.522937+0000","last_clean":"2024-09-18T12:02:11.522937+0000","last_became_active":"2024-09-18T11:45:25.379657+0000","last_became_peered":"2024-09-18T11:45:25.379657+0000","last_unstale":"2024-09-18T12:02:11.522937+0000","last_undegraded":"2024-09-18T12:02:11.522937+0000","last_fullsized":"2024-09-18T12:02:11.522937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:44:38.923827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523379+0000","last_change":"2024-09-18T11:45:25.340204+0000","last_active":"2024-09-18T12:02:11.523379+0000","last_peered":"2024-09-18T12:02:11.523379+0000","last_clean":"2024-09-18T12:02:11.523379+0000","last_became_active":"2024-09-18T11:45:25.338772+0000","last_became_peered":"2024-09-18T11:45:25.338772+0000","last_unstale":"2024-09-18T12:02:11.523379+0000","last_undegraded":"2024-09-18T12:02:11.523379+0000","last_fullsized":"2024-09-18T12:02:11.523379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:18:09.409329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.522520+0000","last_change":"2024-09-18T11:45:25.369783+0000","last_active":"2024-09-18T12:02:11.522520+0000","last_peered":"2024-09-18T12:02:11.522520+0000","last_clean":"2024-09-18T12:02:11.522520+0000","last_became_active":"2024-09-18T11:45:25.369294+0000","last_became_peered":"2024-09-18T11:45:25.369294+0000","last_unstale":"2024-09-18T12:02:11.522520+0000","last_undegraded":"2024-09-18T12:02:11.522520+0000","last_fullsized":"2024-09-18T12:02:11.522520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:20:30.771482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.912743+0000","last_change":"2024-09-18T11:45:25.393624+0000","last_active":"2024-09-18T12:02:10.912743+0000","last_peered":"2024-09-18T12:02:10.912743+0000","last_clean":"2024-09-18T12:02:10.912743+0000","last_became_active":"2024-09-18T11:45:25.393093+0000","last_became_peered":"2024-09-18T11:45:25.393093+0000","last_unstale":"2024-09-18T12:02:10.912743+0000","last_undegraded":"2024-09-18T12:02:10.912743+0000","last_fullsized":"2024-09-18T12:02:10.912743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:17:30.506236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":647,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.911484+0000","last_change":"2024-09-18T11:45:25.382303+0000","last_active":"2024-09-18T12:02:10.911484+0000","last_peered":"2024-09-18T12:02:10.911484+0000","last_clean":"2024-09-18T12:02:10.911484+0000","last_became_active":"2024-09-18T11:45:25.380362+0000","last_became_peered":"2024-09-18T11:45:25.380362+0000","last_unstale":"2024-09-18T12:02:10.911484+0000","last_undegraded":"2024-09-18T12:02:10.911484+0000","last_fullsized":"2024-09-18T12:02:10.911484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:11:31.985540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":645,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.523224+0000","last_change":"2024-09-18T11:45:25.358090+0000","last_active":"2024-09-18T12:02:11.523224+0000","last_peered":"2024-09-18T12:02:11.523224+0000","last_clean":"2024-09-18T12:02:11.523224+0000","last_became_active":"2024-09-18T11:45:25.357477+0000","last_became_peered":"2024-09-18T11:45:25.357477+0000","last_unstale":"2024-09-18T12:02:11.523224+0000","last_undegraded":"2024-09-18T12:02:11.523224+0000","last_fullsized":"2024-09-18T12:02:11.523224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:54:53.792263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":646,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.960338+0000","last_change":"2024-09-18T11:45:25.394821+0000","last_active":"2024-09-18T12:02:10.960338+0000","last_peered":"2024-09-18T12:02:10.960338+0000","last_clean":"2024-09-18T12:02:10.960338+0000","last_became_active":"2024-09-18T11:45:25.394579+0000","last_became_peered":"2024-09-18T11:45:25.394579+0000","last_unstale":"2024-09-18T12:02:10.960338+0000","last_undegraded":"2024-09-18T12:02:10.960338+0000","last_fullsized":"2024-09-18T12:02:10.960338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:59.330658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":638,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-18T12:02:10.914779+0000","last_change":"2024-09-18T11:45:25.405243+0000","last_active":"2024-09-18T12:02:10.914779+0000","last_peered":"2024-09-18T12:02:10.914779+0000","last_clean":"2024-09-18T12:02:10.914779+0000","last_became_active":"2024-09-18T11:45:25.403069+0000","last_became_peered":"2024-09-18T11:45:25.403069+0000","last_unstale":"2024-09-18T12:02:10.914779+0000","last_undegraded":"2024-09-18T12:02:10.914779+0000","last_fullsized":"2024-09-18T12:02:10.914779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:50:14.772996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607789,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315068,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056772,"statfs":{"total":96636764160,"available":96314134528,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.876,"15min":0.873},"min":{"1min":0.56799999999999995,"5min":0.48499999999999999,"15min":0.41499999999999998},"max":{"1min":1.1140000000000001,"5min":1.244,"15min":1.53},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.85199999999999998},"min":{"1min":0.70499999999999996,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0129999999999999,"5min":1.294,"15min":1.454},"last":0.89700000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85599999999999998,"15min":0.879},"min":{"1min":0.58299999999999996,"5min":0.52900000000000003,"15min":0.46500000000000002},"max":{"1min":0.999,"5min":1.2529999999999999,"15min":1.325},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.85999999999999999,"15min":0.86199999999999999},"min":{"1min":0.73799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.98899999999999999,"5min":1.208,"15min":1.2689999999999999},"last":0.78100000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79400000000000004,"15min":0.82199999999999995},"min":{"1min":0.70799999999999996,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":0.95699999999999996,"5min":1.2410000000000001,"15min":2.8900000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89000000000000001,"15min":0.86899999999999999},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.439},"max":{"1min":1.1060000000000001,"5min":1.6060000000000001,"15min":3.0760000000000001},"last":0.83699999999999997}]},{"osd":3,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.79100000000000004,"15min":0.77200000000000002},"min":{"1min":0.61199999999999999,"5min":0.45800000000000002,"15min":0.372},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.6950000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.56799999999999995,"5min":0.439,"15min":0.35299999999999998},"max":{"1min":1.198,"5min":1.204,"15min":1.446},"last":0.68500000000000005}]},{"osd":4,"last update":"Wed Sep 18 12:02:51 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.82799999999999996,"15min":0.86099999999999999},"min":{"1min":0.72899999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.159,"5min":1.159,"15min":1.323},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.98099999999999998,"15min":0.95999999999999996},"min":{"1min":0.82099999999999995,"5min":0.54600000000000004,"15min":0.439},"max":{"1min":1.232,"5min":1.3089999999999999,"15min":1.4670000000000001},"last":0.71199999999999997}]}]},{"osd":4,"up_from":12,"seq":51539607788,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319244,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052596,"statfs":{"total":96636764160,"available":96309858304,"internally_reserved":0,"allocated":9613312,"data_stored":4953859,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.83699999999999997,"15min":0.82899999999999996},"min":{"1min":0.64800000000000002,"5min":0.51800000000000002,"15min":0.47699999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.3120000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.83399999999999996,"15min":0.83099999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.46400000000000002},"max":{"1min":1.131,"5min":1.272,"15min":1.296},"last":0.77100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.504},"max":{"1min":0.92200000000000004,"5min":1.3169999999999999,"15min":1.331},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90500000000000003,"15min":0.85799999999999998},"min":{"1min":0.69199999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3600000000000001},"last":0.72999999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78800000000000003,"15min":0.82399999999999995},"min":{"1min":0.627,"5min":0.56799999999999995,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":1.0329999999999999,"15min":1.294},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.92800000000000005,"15min":0.90300000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.432},"max":{"1min":1.206,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.81899999999999995}]},{"osd":3,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.57999999999999996,"5min":0.48899999999999999,"15min":0.47499999999999998},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.879,"15min":0.89300000000000002},"min":{"1min":0.70599999999999996,"5min":0.5,"15min":0.44600000000000001},"max":{"1min":1.0900000000000001,"5min":1.4019999999999999,"15min":1.496},"last":0.69299999999999995}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.56999999999999995,"5min":0.53500000000000003,"15min":0.40100000000000002},"max":{"1min":1.085,"5min":1.4430000000000001,"15min":2.899},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0289999999999999,"15min":0.97199999999999998},"min":{"1min":0.76900000000000002,"5min":0.69299999999999995,"15min":0.56599999999999995},"max":{"1min":1.28,"5min":1.466,"15min":3.004},"last":0.89000000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607788,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319292,"kb_used_data":9444,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052548,"statfs":{"total":96636764160,"available":96309809152,"internally_reserved":0,"allocated":9670656,"data_stored":5483791,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.75,"15min":0.80500000000000005},"min":{"1min":0.55600000000000005,"5min":0.5,"15min":0.35999999999999999},"max":{"1min":1.2350000000000001,"5min":1.331,"15min":5.1390000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.82099999999999995,"15min":0.876},"min":{"1min":0.66700000000000004,"5min":0.56799999999999995,"15min":0.40999999999999998},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":7.0449999999999999},"last":0.98799999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.79200000000000004,"5min":0.59699999999999998,"15min":0.505},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":7.4880000000000004},"last":0.749},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93400000000000005,"15min":0.91500000000000004},"min":{"1min":0.69799999999999995,"5min":0.55000000000000004,"15min":0.45800000000000002},"max":{"1min":0.97999999999999998,"5min":1.325,"15min":4.9699999999999998},"last":0.95599999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.626,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":0.95499999999999996,"5min":1.419,"15min":5.2080000000000002},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.876,"5min":0.91900000000000004,"15min":0.96599999999999997},"min":{"1min":0.67600000000000005,"5min":0.63600000000000001,"15min":0.441},"max":{"1min":1.1259999999999999,"5min":1.3280000000000001,"15min":7.4180000000000001},"last":0.67900000000000005}]},{"osd":4,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.73099999999999998},"min":{"1min":0.53400000000000003,"5min":0.49299999999999999,"15min":0.39600000000000002},"max":{"1min":0.94199999999999995,"5min":1.2569999999999999,"15min":4.7169999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97699999999999998},"min":{"1min":0.70499999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.361,"5min":1.7150000000000001,"15min":4.6260000000000003},"last":0.77900000000000003}]},{"osd":5,"last update":"Wed Sep 18 12:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.82399999999999995,"15min":0.86499999999999999},"min":{"1min":0.621,"5min":0.54500000000000004,"15min":0.34699999999999998},"max":{"1min":1.159,"5min":1.208,"15min":7.4470000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0740000000000001,"15min":1.1160000000000001},"min":{"1min":0.871,"5min":0.68600000000000005,"15min":0.63200000000000001},"max":{"1min":1.4390000000000001,"5min":1.581,"15min":7.6269999999999998},"last":1.038}]}]},{"osd":2,"up_from":12,"seq":51539607788,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315084,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056756,"statfs":{"total":96636764160,"available":96314118144,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58499999999999996},"min":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.33100000000000002},"max":{"1min":0.98499999999999999,"5min":1.212,"15min":1.212},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.745,"15min":0.72499999999999998},"min":{"1min":0.60999999999999999,"5min":0.48199999999999998,"15min":0.28699999999999998},"max":{"1min":1.02,"5min":1.254,"15min":1.254},"last":0.47499999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.495,"5min":0.41599999999999998,"15min":0.316},"max":{"1min":0.76100000000000001,"5min":0.93500000000000005,"15min":3.0899999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.56999999999999995,"5min":0.49299999999999999,"15min":0.41999999999999998},"max":{"1min":0.84199999999999997,"5min":1.577,"15min":4.1699999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.86599999999999999,"15min":0.89100000000000001},"min":{"1min":0.67800000000000005,"5min":0.52500000000000002,"15min":0.36899999999999999},"max":{"1min":1.0680000000000001,"5min":1.3799999999999999,"15min":2.6389999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.86199999999999999,"15min":0.88400000000000001},"min":{"1min":0.64300000000000002,"5min":0.623,"15min":0.53400000000000003},"max":{"1min":1.0029999999999999,"5min":1.2330000000000001,"15min":4.3310000000000004},"last":0.68000000000000005}]},{"osd":4,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.75900000000000001,"5min":0.67300000000000004,"15min":0.376},"max":{"1min":1.0600000000000001,"5min":1.2270000000000001,"15min":4.2270000000000003},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96999999999999997,"15min":0.94099999999999995},"min":{"1min":0.80800000000000005,"5min":0.69599999999999995,"15min":0.55700000000000005},"max":{"1min":1.0840000000000001,"5min":1.671,"15min":1.706},"last":0.94299999999999995}]},{"osd":5,"last update":"Wed Sep 18 12:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.89900000000000002,"15min":0.92000000000000004},"min":{"1min":0.71499999999999997,"5min":0.56999999999999995,"15min":0.39000000000000001},"max":{"1min":0.95099999999999996,"5min":2.4449999999999998,"15min":2.5960000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.93400000000000005,"15min":0.96199999999999997},"min":{"1min":0.74299999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":0.97099999999999997,"5min":1.4890000000000001,"15min":4.7590000000000003},"last":0.82899999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315144,"kb_used_data":9440,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056696,"statfs":{"total":96636764160,"available":96314056704,"internally_reserved":0,"allocated":9666560,"data_stored":5483786,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55600000000000005,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.38100000000000001,"15min":0.29199999999999998},"max":{"1min":0.66700000000000004,"5min":0.82899999999999996,"15min":3.4900000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66800000000000004,"15min":0.68500000000000005},"min":{"1min":0.52800000000000002,"5min":0.442,"15min":0.27500000000000002},"max":{"1min":0.92700000000000005,"5min":1.0209999999999999,"15min":3.1019999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.67400000000000004},"min":{"1min":0.54900000000000004,"5min":0.443,"15min":0.317},"max":{"1min":0.94799999999999995,"5min":0.97299999999999998,"15min":3.5019999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81899999999999995,"15min":0.79000000000000004},"min":{"1min":0.66700000000000004,"5min":0.50700000000000001,"15min":0.41799999999999998},"max":{"1min":1.0620000000000001,"5min":1.0660000000000001,"15min":3.1779999999999999},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.69899999999999995,"5min":0.57899999999999996,"15min":0.40300000000000002},"max":{"1min":1.1120000000000001,"5min":1.2250000000000001,"15min":1.71},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.88400000000000001,"15min":0.92100000000000004},"min":{"1min":0.67800000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.0229999999999999,"5min":1.26,"15min":4.3719999999999999},"last":0.70599999999999996}]},{"osd":4,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84599999999999997,"15min":0.876},"min":{"1min":0.68600000000000005,"5min":0.58399999999999996,"15min":0.48899999999999999},"max":{"1min":0.91500000000000004,"5min":1.123,"15min":2.4990000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.98899999999999999,"15min":0.93999999999999995},"min":{"1min":0.79500000000000004,"5min":0.70999999999999996,"15min":0.443},"max":{"1min":1.181,"5min":1.262,"15min":1.6679999999999999},"last":0.92300000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.90700000000000003,"15min":0.92400000000000004},"min":{"1min":0.79800000000000004,"5min":0.56100000000000005,"15min":0.504},"max":{"1min":1.0289999999999999,"5min":1.3300000000000001,"15min":2.5840000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.04,"5min":1.0209999999999999,"15min":0.97999999999999998},"min":{"1min":0.84799999999999998,"5min":0.67200000000000004,"15min":0.47399999999999998},"max":{"1min":1.2310000000000001,"5min":1.53,"15min":1.752},"last":1.02}]}]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315076,"kb_used_data":9396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056764,"statfs":{"total":96636764160,"available":96314126336,"internally_reserved":0,"allocated":9621504,"data_stored":4954882,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.191},"max":{"1min":0.69499999999999995,"5min":2.4470000000000001,"15min":3.8849999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73899999999999999,"15min":0.71799999999999997},"min":{"1min":0.54900000000000004,"5min":0.40300000000000002,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65700000000000003,"15min":0.67600000000000005},"min":{"1min":0.53200000000000003,"5min":0.501,"15min":0.23300000000000001},"max":{"1min":0.88100000000000001,"5min":1.3799999999999999,"15min":3.6030000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.871,"15min":0.81699999999999995},"min":{"1min":0.66600000000000004,"5min":0.63100000000000001,"15min":0.30399999999999999},"max":{"1min":1.0569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.89700000000000002}]},{"osd":3,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84899999999999998,"15min":0.84599999999999997},"min":{"1min":0.621,"5min":0.60099999999999998,"15min":0.28100000000000003},"max":{"1min":0.98099999999999998,"5min":2.2440000000000002,"15min":3.7909999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.93799999999999994,"15min":0.92500000000000004},"min":{"1min":0.69799999999999995,"5min":0.65800000000000003,"15min":0.313},"max":{"1min":1.248,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.0089999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.002,"15min":0.95299999999999996},"min":{"1min":0.82199999999999995,"5min":0.57199999999999995,"15min":0.40799999999999997},"max":{"1min":1.085,"5min":2.8530000000000002,"15min":3.6709999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96899999999999997,"15min":0.996},"min":{"1min":0.67900000000000005,"5min":0.67400000000000004,"15min":0.33700000000000002},"max":{"1min":1.175,"5min":2.726,"15min":3.5270000000000001},"last":1.111}]},{"osd":5,"last update":"Wed Sep 18 12:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88800000000000001},"min":{"1min":0.80800000000000005,"5min":0.66700000000000004,"15min":0.36799999999999999},"max":{"1min":1.0289999999999999,"5min":1.389,"15min":3.6469999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.98899999999999999,"15min":0.97499999999999998},"min":{"1min":0.71899999999999997,"5min":0.70099999999999996,"15min":0.39500000000000002},"max":{"1min":0.99299999999999999,"5min":2.7429999999999999,"15min":4.3280000000000003},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:43.823 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.823 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.823 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.823 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.823 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.823 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.823 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.823 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.824 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.824 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.824 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.824 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.824 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.824 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.824 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.824 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.825 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T11:44:20.059736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=44, tm_sec=20, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.826 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.827 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.827 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.827 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.827 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.827 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.827 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:03:43.827 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T12:04:03.828 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:03.940 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient: get_monmap_and_config 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient: build_initial_monmap 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient: monmap: 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:04:03.941822+0000 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:04:03.941822+0000 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:04:03.941 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:04:03.942 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:04:03.942 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 1 Processor -- start 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 1 -- start start 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient: init 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient: _reopen_session rank -1 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f5bb80a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f5bb80a0750 addr v1:172.21.15.129:6789/0 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f5bb81385b0 addr v1:172.21.15.112:6789/0 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb8097270 con 0x7f5bb81385b0 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80a3170 con 0x7f5bb80a0750 2024-09-18T12:04:03.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb8096cb0 con 0x7f5bb80a0380 2024-09-18T12:04:03.944 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient(hunting): _renew_subs 2024-09-18T12:04:03.944 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb7fff700 10 monclient(hunting): authenticate will time out at 2015.783047s 2024-09-18T12:04:03.944 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5bb81385b0 0x7f5bb813a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:57776/0 (socket says 172.21.15.112:57776) 2024-09-18T12:04:03.944 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb77fe700 1 -- 172.21.15.112:0/2646568905 learned_addr learned my addr 172.21.15.112:0/2646568905 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:04:03.944 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.936+0000 7f5bb6ffd700 1 --1- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6790/0 conn(0x7f5bb80a0380 0x7f5bb8097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:60642/0 (socket says 172.21.15.112:60642) 2024-09-18T12:04:03.944 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb77fe700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6789/0 conn(0x7f5bb81385b0 legacy=0x7f5bb813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb67fc700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.129:6789/0 conn(0x7f5bb80a0750 legacy=0x7f5bb80a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb6ffd700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6790/0 conn(0x7f5bb80a0380 legacy=0x7f5bb8097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268376014 0 0) 0x7f5bb80a3170 con 0x7f5bb80a0750 2024-09-18T12:04:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): my global_id is 144421 2024-09-18T12:04:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba0003570 con 0x7f5bb80a0750 2024-09-18T12:04:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590336541 0 0) 0x7f5bb8097270 con 0x7f5bb81385b0 2024-09-18T12:04:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): my global_id is 164238 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb80a3170 con 0x7f5bb81385b0 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542033838 0 0) 0x7f5bb8096cb0 con 0x7f5bb80a0380 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): my global_id is 154334 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb8097270 con 0x7f5bb80a0380 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4049644234 0 0) 0x7f5bb80a3170 con 0x7f5bb81385b0 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6790/0 conn(0x7f5bb80a0380 legacy=0x7f5bb8097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.129:6789/0 conn(0x7f5bb80a0750 legacy=0x7f5bb80a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:04:03.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 monclient: found mon.noname-a 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80a1420 con 0x7f5bb81385b0 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: _finish_auth 0 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5bb0001ca0 con 0x7f5bb81385b0 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb7fff700 5 monclient: authenticate success, global_id 164238 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: dump: 2024-09-18T12:04:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb77fe700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6789/0 conn(0x7f5bb81385b0 legacy=0x7f5bb813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb77fe700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6789/0 conn(0x7f5bb81385b0 legacy=0x7f5bb813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb77fe700 1 --1- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6789/0 conn(0x7f5bb81385b0 0x7f5bb813a990 :-1 s=OPENED pgs=745 cs=1 l=1).handle_message read tag failed 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb77fe700 1 --1- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6789/0 conn(0x7f5bb81385b0 0x7f5bb813a990 :-1 s=OPENED pgs=745 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:04:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: ms_handle_reset current mon v1:172.21.15.112:6789/0 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: _reopen_session rank -1 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6789/0 conn(0x7f5bb81385b0 legacy=0x7f5bb813a990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): picked mon.a con 0x7f5ba0004fc0 addr v1:172.21.15.112:6789/0 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): picked mon.c con 0x7f5bb80a0380 addr v1:172.21.15.112:6790/0 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): picked mon.b con 0x7f5bb80a0750 addr v1:172.21.15.129:6789/0 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bb80a3170 con 0x7f5ba0004fc0 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5ba0003570 con 0x7f5bb80a0750 2024-09-18T12:04:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bb8097270 con 0x7f5bb80a0380 2024-09-18T12:04:03.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): _renew_subs 2024-09-18T12:04:03.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.940+0000 7f5bb5ffb700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.112:6789/0 2024-09-18T12:04:03.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb67fc700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6790/0 conn(0x7f5bb80a0380 legacy=0x7f5ba0003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb6ffd700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6789/0 conn(0x7f5ba0004fc0 legacy=0x7f5ba00037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478979498 0 0) 0x7f5bb8097270 con 0x7f5bb80a0380 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.112:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f5ba000f1a0 con 0x7f5bb80a0380 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453793234 0 0) 0x7f5bb80a3170 con 0x7f5ba0004fc0 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.112:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f5bb8097270 con 0x7f5ba0004fc0 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb77fe700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.129:6789/0 conn(0x7f5bb80a0750 legacy=0x7f5ba000da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149246016 0 0) 0x7f5ba0003570 con 0x7f5bb80a0750 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.129:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f5bb80a3170 con 0x7f5bb80a0750 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1154004651 0 0) 0x7f5ba000f1a0 con 0x7f5bb80a0380 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.129:6789/0 conn(0x7f5bb80a0750 legacy=0x7f5ba000da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6789/0 conn(0x7f5ba0004fc0 legacy=0x7f5ba00037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 monclient: found mon.c 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f5bb8096cb0 con 0x7f5bb80a0380 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient: _finish_auth 0 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5ba80020a0 con 0x7f5bb80a0380 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:03.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient: dump: 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 1 -- 172.21.15.112:0/2646568905 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ba8002200 con 0x7f5bb80a0380 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 10 monclient: get_monmap_and_config success 2024-09-18T12:04:03.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 4 set_mon_vals no callback set 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 10 monclient: shutdown 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 1 -- 172.21.15.112:0/2646568905 >> v1:172.21.15.112:6790/0 conn(0x7f5bb80a0380 legacy=0x7f5ba0003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 1 -- 172.21.15.112:0/2646568905 shutdown_connections 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 1 -- 172.21.15.112:0/2646568905 >> 172.21.15.112:0/2646568905 conn(0x7f5bb80a5990 msgr2=0x7f5bb80a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 1 -- 172.21.15.112:0/2646568905 shutdown_connections 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.944+0000 7f5bb7fff700 1 -- 172.21.15.112:0/2646568905 wait complete. 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient: build_initial_monmap 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient: monmap: 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:03.963 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 librados: starting msgr at 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 librados: starting objecter 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 Processor -- start 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 -- start start 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 librados: setting wanted keys 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 librados: calling monclient init 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient: init 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient: _reopen_session rank -1 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f5ba0004fc0 addr v1:172.21.15.129:6789/0 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f5bb80a0380 addr v1:172.21.15.112:6789/0 2024-09-18T12:04:03.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f5bb80a0750 addr v1:172.21.15.112:6790/0 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb8122610 con 0x7f5bb80a0380 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb813d9a0 con 0x7f5ba0004fc0 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb8122330 con 0x7f5bb80a0750 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient(hunting): _renew_subs 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb7fff700 10 monclient(hunting): authenticate will time out at 2015.793753s 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb77fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7f5bb80a0750 0x7f5bb8149c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:60660/0 (socket says 172.21.15.112:60660) 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.948+0000 7f5bb77fe700 1 -- 172.21.15.112:0/1278626317 learned_addr learned my addr 172.21.15.112:0/1278626317 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb67fc700 1 -- 172.21.15.112:0/1278626317 >> v1:172.21.15.112:6789/0 conn(0x7f5bb80a0380 legacy=0x7f5bb81464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 client.?.objecter ms_handle_connect 0x7f5bb80a0380 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb77fe700 1 -- 172.21.15.112:0/1278626317 >> v1:172.21.15.112:6790/0 conn(0x7f5bb80a0750 legacy=0x7f5bb8149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 client.?.objecter ms_handle_connect 0x7f5bb80a0750 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb6ffd700 1 -- 172.21.15.112:0/1278626317 >> v1:172.21.15.129:6789/0 conn(0x7f5ba0004fc0 legacy=0x7f5bb813d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:03.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213427854 0 0) 0x7f5bb8122610 con 0x7f5bb80a0380 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): my global_id is 164241 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba4004f10 con 0x7f5bb80a0380 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 client.?.objecter ms_handle_connect 0x7f5ba0004fc0 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746116728 0 0) 0x7f5bb8122330 con 0x7f5bb80a0750 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): my global_id is 154337 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb8122610 con 0x7f5bb80a0750 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178388070 0 0) 0x7f5bb813d9a0 con 0x7f5ba0004fc0 2024-09-18T12:04:03.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): my global_id is 144424 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb8122330 con 0x7f5ba0004fc0 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2876593020 0 0) 0x7f5ba4004f10 con 0x7f5bb80a0380 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb813d9a0 con 0x7f5bb80a0380 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5ba8001f00 con 0x7f5bb80a0380 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): dump: 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:03.967 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2158609315 0 0) 0x7f5bb8122610 con 0x7f5bb80a0750 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ba4004f10 con 0x7f5bb80a0750 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5bb0001a60 con 0x7f5bb80a0750 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): dump: 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:03.968 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2488126185 0 0) 0x7f5bb8122330 con 0x7f5ba0004fc0 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb8122610 con 0x7f5ba0004fc0 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5bac0016a0 con 0x7f5ba0004fc0 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient(hunting): dump: 2024-09-18T12:04:03.969 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3193775724 0 0) 0x7f5bb813d9a0 con 0x7f5bb80a0380 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 >> v1:172.21.15.112:6790/0 conn(0x7f5bb80a0750 legacy=0x7f5bb8149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 >> v1:172.21.15.129:6789/0 conn(0x7f5ba0004fc0 legacy=0x7f5bb813d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient: _finish_hunting 0 2024-09-18T12:04:03.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 monclient: found mon.a 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb81200a0 con 0x7f5bb80a0380 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient: _finish_auth 0 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 5 monclient: authenticate success, global_id 164241 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 10 monclient: _renew_subs 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb813e660 con 0x7f5bb80a0380 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 10 client.164241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 10 monclient: _renew_subs 2024-09-18T12:04:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb8121600 con 0x7f5bb80a0380 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 1 librados: init done 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ba8002370 con 0x7f5bb80a0380 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7f5ba8002680 con 0x7f5bb80a0380 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb57fa700 4 set_mon_vals no callback set 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 monclient: dump: 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7f5ba80028e0 con 0x7f5bb80a0380 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:04:03.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b980050b0 con 0x7f5bb80a0380 2024-09-18T12:04:03.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7f5ba8043ac0 con 0x7f5bb80a0380 2024-09-18T12:04:03.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 10 client.164241.objecter ms_dispatch 0x7f5bb8000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:04:03.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 3 client.164241.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:04:03.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.952+0000 7f5b9effd700 3 client.164241.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:04:03.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.956+0000 7f5b9effd700 20 client.164241.objecter dump_active .. 0 homeless 2024-09-18T12:04:03.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.956+0000 7f5bb6ffd700 10 monclient: get_auth_request con 0x7f5ba4038620 auth_method 0 2024-09-18T12:04:03.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.956+0000 7f5b9effd700 10 client.164241.objecter ms_handle_connect 0x7f5ba4038620 2024-09-18T12:04:03.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.960+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7f5ba8014620 con 0x7f5bb80a0380 2024-09-18T12:04:03.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.960+0000 7f5b9effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:03.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:03.960+0000 7f5b9effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:04:04.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.124+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b98000b90 con 0x7f5ba4038620 2024-09-18T12:04:04.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5b9effd700 1 -- 172.21.15.112:0/1278626317 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134217 (unknown 2965378022 0 1787803184) 0x7f5b98000b90 con 0x7f5ba4038620 2024-09-18T12:04:04.138 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T12:04:04.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 10 librados: watch_flush enter 2024-09-18T12:04:04.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 10 librados: watch_flush exit 2024-09-18T12:04:04.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 20 client.164241.objecter shutdown clearing up homeless session... 2024-09-18T12:04:04.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 10 client.164241.objecter successfully canceled tick 2024-09-18T12:04:04.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 >> v1:172.21.15.129:6812/20806 conn(0x7f5ba4038620 legacy=0x7f5ba403aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 10 monclient: shutdown 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 >> v1:172.21.15.112:6789/0 conn(0x7f5bb80a0380 legacy=0x7f5bb81464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 shutdown_connections 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 >> 172.21.15.112:0/1278626317 conn(0x7f5bb80a5990 msgr2=0x7f5bb80a4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 shutdown_connections 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 1 -- 172.21.15.112:0/1278626317 wait complete. 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:04.136+0000 7f5bb7fff700 1 librados: shutdown 2024-09-18T12:04:04.141 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T12:04:04.152 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1286,"stamp":"2024-09-18T12:04:02.388976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1898796,"kb_used_data":56444,"kb_used_omap":9,"kb_used_meta":1842294,"kb_avail":564332244,"statfs":{"total":579820584960,"available":577876217856,"internally_reserved":0,"allocated":57798656,"data_stored":30786034,"data_compressed":320902,"data_compressed_allocated":876544,"data_compressed_original":2744320,"omap_allocated":10224,"internal_metadata":1886509072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004257"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.914458+0000","last_change":"2024-09-18T11:45:25.374991+0000","last_active":"2024-09-18T12:02:11.914458+0000","last_peered":"2024-09-18T12:02:11.914458+0000","last_clean":"2024-09-18T12:02:11.914458+0000","last_became_active":"2024-09-18T11:45:25.373642+0000","last_became_peered":"2024-09-18T11:45:25.373642+0000","last_unstale":"2024-09-18T12:02:11.914458+0000","last_undegraded":"2024-09-18T12:02:11.914458+0000","last_fullsized":"2024-09-18T12:02:11.914458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.015861+0000","last_change":"2024-09-18T12:03:58.015792+0000","last_active":"2024-09-18T12:03:58.015861+0000","last_peered":"2024-09-18T12:03:58.015861+0000","last_clean":"2024-09-18T12:03:58.015861+0000","last_became_active":"2024-09-18T11:45:25.328032+0000","last_became_peered":"2024-09-18T11:45:25.328032+0000","last_unstale":"2024-09-18T12:03:58.015861+0000","last_undegraded":"2024-09-18T12:03:58.015861+0000","last_fullsized":"2024-09-18T12:03:58.015861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:58.015589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:58.015589+0000","last_clean_scrub_stamp":"2024-09-18T12:03:58.015589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:56:03.382870+0000","scrub_duration":0.0031503450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.788584+0000","last_change":"2024-09-18T12:03:49.788515+0000","last_active":"2024-09-18T12:03:49.788584+0000","last_peered":"2024-09-18T12:03:49.788584+0000","last_clean":"2024-09-18T12:03:49.788584+0000","last_became_active":"2024-09-18T11:45:25.305872+0000","last_became_peered":"2024-09-18T11:45:25.305872+0000","last_unstale":"2024-09-18T12:03:49.788584+0000","last_undegraded":"2024-09-18T12:03:49.788584+0000","last_fullsized":"2024-09-18T12:03:49.788584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:49.788307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:49.788307+0000","last_clean_scrub_stamp":"2024-09-18T12:03:49.788307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:25:23.899899+0000","scrub_duration":0.0033892200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:56.980139+0000","last_change":"2024-09-18T12:03:56.980022+0000","last_active":"2024-09-18T12:03:56.980139+0000","last_peered":"2024-09-18T12:03:56.980139+0000","last_clean":"2024-09-18T12:03:56.980139+0000","last_became_active":"2024-09-18T11:45:25.411133+0000","last_became_peered":"2024-09-18T11:45:25.411133+0000","last_unstale":"2024-09-18T12:03:56.980139+0000","last_undegraded":"2024-09-18T12:03:56.980139+0000","last_fullsized":"2024-09-18T12:03:56.980139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:56.979808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:56.979808+0000","last_clean_scrub_stamp":"2024-09-18T12:03:56.979808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:08:59.245327+0000","scrub_duration":0.003648044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":654,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.184043+0000","last_change":"2024-09-18T12:03:53.183942+0000","last_active":"2024-09-18T12:03:53.184043+0000","last_peered":"2024-09-18T12:03:53.184043+0000","last_clean":"2024-09-18T12:03:53.184043+0000","last_became_active":"2024-09-18T11:45:25.399898+0000","last_became_peered":"2024-09-18T11:45:25.399898+0000","last_unstale":"2024-09-18T12:03:53.184043+0000","last_undegraded":"2024-09-18T12:03:53.184043+0000","last_fullsized":"2024-09-18T12:03:53.184043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:53.183699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:53.183699+0000","last_clean_scrub_stamp":"2024-09-18T12:03:53.183699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:03:01.050741+0000","scrub_duration":0.0035415669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:59.028630+0000","last_change":"2024-09-18T12:03:59.028574+0000","last_active":"2024-09-18T12:03:59.028630+0000","last_peered":"2024-09-18T12:03:59.028630+0000","last_clean":"2024-09-18T12:03:59.028630+0000","last_became_active":"2024-09-18T11:45:25.396975+0000","last_became_peered":"2024-09-18T11:45:25.396975+0000","last_unstale":"2024-09-18T12:03:59.028630+0000","last_undegraded":"2024-09-18T12:03:59.028630+0000","last_fullsized":"2024-09-18T12:03:59.028630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:59.028376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:59.028376+0000","last_clean_scrub_stamp":"2024-09-18T12:03:59.028376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:00:50.746697+0000","scrub_duration":0.0034269259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":648,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.911447+0000","last_change":"2024-09-18T11:45:25.341093+0000","last_active":"2024-09-18T12:02:11.911447+0000","last_peered":"2024-09-18T12:02:11.911447+0000","last_clean":"2024-09-18T12:02:11.911447+0000","last_became_active":"2024-09-18T11:45:25.340726+0000","last_became_peered":"2024-09-18T11:45:25.340726+0000","last_unstale":"2024-09-18T12:02:11.911447+0000","last_undegraded":"2024-09-18T12:02:11.911447+0000","last_fullsized":"2024-09-18T12:02:11.911447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":657,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:46.396477+0000","last_change":"2024-09-18T12:03:46.396412+0000","last_active":"2024-09-18T12:03:46.396477+0000","last_peered":"2024-09-18T12:03:46.396477+0000","last_clean":"2024-09-18T12:03:46.396477+0000","last_became_active":"2024-09-18T11:45:25.408464+0000","last_became_peered":"2024-09-18T11:45:25.408464+0000","last_unstale":"2024-09-18T12:03:46.396477+0000","last_undegraded":"2024-09-18T12:03:46.396477+0000","last_fullsized":"2024-09-18T12:03:46.396477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:46.396196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:46.396196+0000","last_clean_scrub_stamp":"2024-09-18T12:03:46.396196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:46:03.414527+0000","scrub_duration":0.0035477960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.670831+0000","last_change":"2024-09-18T12:03:58.670753+0000","last_active":"2024-09-18T12:03:58.670831+0000","last_peered":"2024-09-18T12:03:58.670831+0000","last_clean":"2024-09-18T12:03:58.670831+0000","last_became_active":"2024-09-18T11:45:25.343097+0000","last_became_peered":"2024-09-18T11:45:25.343097+0000","last_unstale":"2024-09-18T12:03:58.670831+0000","last_undegraded":"2024-09-18T12:03:58.670831+0000","last_fullsized":"2024-09-18T12:03:58.670831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:58.670528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:58.670528+0000","last_clean_scrub_stamp":"2024-09-18T12:03:58.670528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:29:07.343349+0000","scrub_duration":0.0032734999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:47.880890+0000","last_change":"2024-09-18T12:03:47.880826+0000","last_active":"2024-09-18T12:03:47.880890+0000","last_peered":"2024-09-18T12:03:47.880890+0000","last_clean":"2024-09-18T12:03:47.880890+0000","last_became_active":"2024-09-18T11:45:25.377194+0000","last_became_peered":"2024-09-18T11:45:25.377194+0000","last_unstale":"2024-09-18T12:03:47.880890+0000","last_undegraded":"2024-09-18T12:03:47.880890+0000","last_fullsized":"2024-09-18T12:03:47.880890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:47.880598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:47.880598+0000","last_clean_scrub_stamp":"2024-09-18T12:03:47.880598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:53:32.792745+0000","scrub_duration":0.0034949949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.914639+0000","last_change":"2024-09-18T11:45:25.346404+0000","last_active":"2024-09-18T12:02:11.914639+0000","last_peered":"2024-09-18T12:02:11.914639+0000","last_clean":"2024-09-18T12:02:11.914639+0000","last_became_active":"2024-09-18T11:45:25.341682+0000","last_became_peered":"2024-09-18T11:45:25.341682+0000","last_unstale":"2024-09-18T12:02:11.914639+0000","last_undegraded":"2024-09-18T12:02:11.914639+0000","last_fullsized":"2024-09-18T12:02:11.914639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.914534+0000","last_change":"2024-09-18T11:45:25.383080+0000","last_active":"2024-09-18T12:02:11.914534+0000","last_peered":"2024-09-18T12:02:11.914534+0000","last_clean":"2024-09-18T12:02:11.914534+0000","last_became_active":"2024-09-18T11:45:25.377663+0000","last_became_peered":"2024-09-18T11:45:25.377663+0000","last_unstale":"2024-09-18T12:02:11.914534+0000","last_undegraded":"2024-09-18T12:02:11.914534+0000","last_fullsized":"2024-09-18T12:02:11.914534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":660,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:46.802350+0000","last_change":"2024-09-18T12:03:46.802290+0000","last_active":"2024-09-18T12:03:46.802350+0000","last_peered":"2024-09-18T12:03:46.802350+0000","last_clean":"2024-09-18T12:03:46.802350+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T12:03:46.802350+0000","last_undegraded":"2024-09-18T12:03:46.802350+0000","last_fullsized":"2024-09-18T12:03:46.802350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:46.802087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:46.802087+0000","last_clean_scrub_stamp":"2024-09-18T12:03:46.802087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:05:34.411795+0000","scrub_duration":0.003807094,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":657,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.304300+0000","last_change":"2024-09-18T12:03:49.304241+0000","last_active":"2024-09-18T12:03:49.304300+0000","last_peered":"2024-09-18T12:03:49.304300+0000","last_clean":"2024-09-18T12:03:49.304300+0000","last_became_active":"2024-09-18T11:45:25.405515+0000","last_became_peered":"2024-09-18T11:45:25.405515+0000","last_unstale":"2024-09-18T12:03:49.304300+0000","last_undegraded":"2024-09-18T12:03:49.304300+0000","last_fullsized":"2024-09-18T12:03:49.304300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:49.304048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:49.304048+0000","last_clean_scrub_stamp":"2024-09-18T12:03:49.304048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:51:38.831116+0000","scrub_duration":0.0034724690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":669,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.640895+0000","last_change":"2024-09-18T12:03:45.640825+0000","last_active":"2024-09-18T12:03:45.640895+0000","last_peered":"2024-09-18T12:03:45.640895+0000","last_clean":"2024-09-18T12:03:45.640895+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T12:03:45.640895+0000","last_undegraded":"2024-09-18T12:03:45.640895+0000","last_fullsized":"2024-09-18T12:03:45.640895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:45.640579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:45.640579+0000","last_clean_scrub_stamp":"2024-09-18T12:03:45.640579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:27:16.770756+0000","scrub_duration":0.0037749910000000001,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.825739+0000","last_change":"2024-09-18T12:03:53.825666+0000","last_active":"2024-09-18T12:03:53.825739+0000","last_peered":"2024-09-18T12:03:53.825739+0000","last_clean":"2024-09-18T12:03:53.825739+0000","last_became_active":"2024-09-18T11:45:25.305686+0000","last_became_peered":"2024-09-18T11:45:25.305686+0000","last_unstale":"2024-09-18T12:03:53.825739+0000","last_undegraded":"2024-09-18T12:03:53.825739+0000","last_fullsized":"2024-09-18T12:03:53.825739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:53.825473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:53.825473+0000","last_clean_scrub_stamp":"2024-09-18T12:03:53.825473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:07:07.289345+0000","scrub_duration":0.0035215210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":669,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.849844+0000","last_change":"2024-09-18T12:03:45.849749+0000","last_active":"2024-09-18T12:03:45.849844+0000","last_peered":"2024-09-18T12:03:45.849844+0000","last_clean":"2024-09-18T12:03:45.849844+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T12:03:45.849844+0000","last_undegraded":"2024-09-18T12:03:45.849844+0000","last_fullsized":"2024-09-18T12:03:45.849844+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:45.849511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:45.849511+0000","last_clean_scrub_stamp":"2024-09-18T12:03:45.849511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:48.324946+0000","scrub_duration":0.0035045440000000001,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"699'8","reported_seq":672,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:54.702984+0000","last_change":"2024-09-18T12:03:54.702924+0000","last_active":"2024-09-18T12:03:54.702984+0000","last_peered":"2024-09-18T12:03:54.702984+0000","last_clean":"2024-09-18T12:03:54.702984+0000","last_became_active":"2024-09-18T11:45:25.403586+0000","last_became_peered":"2024-09-18T11:45:25.403586+0000","last_unstale":"2024-09-18T12:03:54.702984+0000","last_undegraded":"2024-09-18T12:03:54.702984+0000","last_fullsized":"2024-09-18T12:03:54.702984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'8","last_scrub_stamp":"2024-09-18T12:03:54.702747+0000","last_deep_scrub":"699'8","last_deep_scrub_stamp":"2024-09-18T12:03:54.702747+0000","last_clean_scrub_stamp":"2024-09-18T12:03:54.702747+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:25:10.453350+0000","scrub_duration":0.0064599619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":665,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:47.165243+0000","last_change":"2024-09-18T12:03:47.165176+0000","last_active":"2024-09-18T12:03:47.165243+0000","last_peered":"2024-09-18T12:03:47.165243+0000","last_clean":"2024-09-18T12:03:47.165243+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T12:03:47.165243+0000","last_undegraded":"2024-09-18T12:03:47.165243+0000","last_fullsized":"2024-09-18T12:03:47.165243+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:47.164959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:47.164959+0000","last_clean_scrub_stamp":"2024-09-18T12:03:47.164959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:10:06.246346+0000","scrub_duration":0.0033963439999999999,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:55.532841+0000","last_change":"2024-09-18T12:03:55.532787+0000","last_active":"2024-09-18T12:03:55.532841+0000","last_peered":"2024-09-18T12:03:55.532841+0000","last_clean":"2024-09-18T12:03:55.532841+0000","last_became_active":"2024-09-18T11:45:25.367362+0000","last_became_peered":"2024-09-18T11:45:25.367362+0000","last_unstale":"2024-09-18T12:03:55.532841+0000","last_undegraded":"2024-09-18T12:03:55.532841+0000","last_fullsized":"2024-09-18T12:03:55.532841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:55.532635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:55.532635+0000","last_clean_scrub_stamp":"2024-09-18T12:03:55.532635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:45:02.887722+0000","scrub_duration":0.0026179189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"699'74","reported_seq":756,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.712591+0000","last_change":"2024-09-18T12:03:53.712497+0000","last_active":"2024-09-18T12:03:53.712591+0000","last_peered":"2024-09-18T12:03:53.712591+0000","last_clean":"2024-09-18T12:03:53.712591+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T12:03:53.712591+0000","last_undegraded":"2024-09-18T12:03:53.712591+0000","last_fullsized":"2024-09-18T12:03:53.712591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'74","last_scrub_stamp":"2024-09-18T12:03:53.712046+0000","last_deep_scrub":"699'74","last_deep_scrub_stamp":"2024-09-18T12:03:53.712046+0000","last_clean_scrub_stamp":"2024-09-18T12:03:53.712046+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:50.861870+0000","scrub_duration":0.0099226230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"699'2","reported_seq":667,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.137599+0000","last_change":"2024-09-18T12:03:45.137534+0000","last_active":"2024-09-18T12:03:45.137599+0000","last_peered":"2024-09-18T12:03:45.137599+0000","last_clean":"2024-09-18T12:03:45.137599+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T12:03:45.137599+0000","last_undegraded":"2024-09-18T12:03:45.137599+0000","last_fullsized":"2024-09-18T12:03:45.137599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'2","last_scrub_stamp":"2024-09-18T12:03:45.137311+0000","last_deep_scrub":"699'2","last_deep_scrub_stamp":"2024-09-18T12:03:45.137311+0000","last_clean_scrub_stamp":"2024-09-18T12:03:45.137311+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:20:51.917782+0000","scrub_duration":0.006619219,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:56.760649+0000","last_change":"2024-09-18T12:03:56.760584+0000","last_active":"2024-09-18T12:03:56.760649+0000","last_peered":"2024-09-18T12:03:56.760649+0000","last_clean":"2024-09-18T12:03:56.760649+0000","last_became_active":"2024-09-18T11:45:25.406430+0000","last_became_peered":"2024-09-18T11:45:25.406430+0000","last_unstale":"2024-09-18T12:03:56.760649+0000","last_undegraded":"2024-09-18T12:03:56.760649+0000","last_fullsized":"2024-09-18T12:03:56.760649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:56.760370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:56.760370+0000","last_clean_scrub_stamp":"2024-09-18T12:03:56.760370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:49:03.599476+0000","scrub_duration":0.0033169749999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"699'3","reported_seq":672,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:43.797405+0000","last_change":"2024-09-18T12:03:43.797302+0000","last_active":"2024-09-18T12:03:43.797405+0000","last_peered":"2024-09-18T12:03:43.797405+0000","last_clean":"2024-09-18T12:03:43.797405+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T12:03:43.797405+0000","last_undegraded":"2024-09-18T12:03:43.797405+0000","last_fullsized":"2024-09-18T12:03:43.797405+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'3","last_scrub_stamp":"2024-09-18T12:03:43.796989+0000","last_deep_scrub":"699'3","last_deep_scrub_stamp":"2024-09-18T12:03:43.796989+0000","last_clean_scrub_stamp":"2024-09-18T12:03:43.796989+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:48:51.634241+0000","scrub_duration":0.0066051479999999999,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.607421+0000","last_change":"2024-09-18T12:03:49.607335+0000","last_active":"2024-09-18T12:03:49.607421+0000","last_peered":"2024-09-18T12:03:49.607421+0000","last_clean":"2024-09-18T12:03:49.607421+0000","last_became_active":"2024-09-18T11:45:25.363235+0000","last_became_peered":"2024-09-18T11:45:25.363235+0000","last_unstale":"2024-09-18T12:03:49.607421+0000","last_undegraded":"2024-09-18T12:03:49.607421+0000","last_fullsized":"2024-09-18T12:03:49.607421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:49.607001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:49.607001+0000","last_clean_scrub_stamp":"2024-09-18T12:03:49.607001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:18:26.489548+0000","scrub_duration":0.003314993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":660,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:47.787279+0000","last_change":"2024-09-18T12:03:47.787216+0000","last_active":"2024-09-18T12:03:47.787279+0000","last_peered":"2024-09-18T12:03:47.787279+0000","last_clean":"2024-09-18T12:03:47.787279+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T12:03:47.787279+0000","last_undegraded":"2024-09-18T12:03:47.787279+0000","last_fullsized":"2024-09-18T12:03:47.787279+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:47.787025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:47.787025+0000","last_clean_scrub_stamp":"2024-09-18T12:03:47.787025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:26:47.388077+0000","scrub_duration":0.0031932520000000002,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":657,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:47.372331+0000","last_change":"2024-09-18T12:03:47.372270+0000","last_active":"2024-09-18T12:03:47.372331+0000","last_peered":"2024-09-18T12:03:47.372331+0000","last_clean":"2024-09-18T12:03:47.372331+0000","last_became_active":"2024-09-18T11:45:25.356178+0000","last_became_peered":"2024-09-18T11:45:25.356178+0000","last_unstale":"2024-09-18T12:03:47.372331+0000","last_undegraded":"2024-09-18T12:03:47.372331+0000","last_fullsized":"2024-09-18T12:03:47.372331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:47.372089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:47.372089+0000","last_clean_scrub_stamp":"2024-09-18T12:03:47.372089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:45:31.859250+0000","scrub_duration":0.0033298849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":669,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:42.617427+0000","last_change":"2024-09-18T12:03:42.617368+0000","last_active":"2024-09-18T12:03:42.617427+0000","last_peered":"2024-09-18T12:03:42.617427+0000","last_clean":"2024-09-18T12:03:42.617427+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T12:03:42.617427+0000","last_undegraded":"2024-09-18T12:03:42.617427+0000","last_fullsized":"2024-09-18T12:03:42.617427+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:42.617180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:42.617180+0000","last_clean_scrub_stamp":"2024-09-18T12:03:42.617180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:30:43.297276+0000","scrub_duration":0.0031515900000000001,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.927805+0000","last_change":"2024-09-18T12:03:51.927734+0000","last_active":"2024-09-18T12:03:51.927805+0000","last_peered":"2024-09-18T12:03:51.927805+0000","last_clean":"2024-09-18T12:03:51.927805+0000","last_became_active":"2024-09-18T11:45:25.396428+0000","last_became_peered":"2024-09-18T11:45:25.396428+0000","last_unstale":"2024-09-18T12:03:51.927805+0000","last_undegraded":"2024-09-18T12:03:51.927805+0000","last_fullsized":"2024-09-18T12:03:51.927805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:51.927549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:51.927549+0000","last_clean_scrub_stamp":"2024-09-18T12:03:51.927549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:05:50.993805+0000","scrub_duration":0.003637361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":648,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.911854+0000","last_change":"2024-09-18T11:45:25.399933+0000","last_active":"2024-09-18T12:02:11.911854+0000","last_peered":"2024-09-18T12:02:11.911854+0000","last_clean":"2024-09-18T12:02:11.911854+0000","last_became_active":"2024-09-18T11:45:25.399695+0000","last_became_peered":"2024-09-18T11:45:25.399695+0000","last_unstale":"2024-09-18T12:02:11.911854+0000","last_undegraded":"2024-09-18T12:02:11.911854+0000","last_fullsized":"2024-09-18T12:02:11.911854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:57.757282+0000","last_change":"2024-09-18T12:03:57.757190+0000","last_active":"2024-09-18T12:03:57.757282+0000","last_peered":"2024-09-18T12:03:57.757282+0000","last_clean":"2024-09-18T12:03:57.757282+0000","last_became_active":"2024-09-18T11:45:25.386932+0000","last_became_peered":"2024-09-18T11:45:25.386932+0000","last_unstale":"2024-09-18T12:03:57.757282+0000","last_undegraded":"2024-09-18T12:03:57.757282+0000","last_fullsized":"2024-09-18T12:03:57.757282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:57.756872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:57.756872+0000","last_clean_scrub_stamp":"2024-09-18T12:03:57.756872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:43:49.029283+0000","scrub_duration":0.0037374650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":644,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.914656+0000","last_change":"2024-09-18T11:45:25.403356+0000","last_active":"2024-09-18T12:02:11.914656+0000","last_peered":"2024-09-18T12:02:11.914656+0000","last_clean":"2024-09-18T12:02:11.914656+0000","last_became_active":"2024-09-18T11:45:25.401979+0000","last_became_peered":"2024-09-18T11:45:25.401979+0000","last_unstale":"2024-09-18T12:02:11.914656+0000","last_undegraded":"2024-09-18T12:02:11.914656+0000","last_fullsized":"2024-09-18T12:02:11.914656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:46.660539+0000","last_change":"2024-09-18T12:03:46.660473+0000","last_active":"2024-09-18T12:03:46.660539+0000","last_peered":"2024-09-18T12:03:46.660539+0000","last_clean":"2024-09-18T12:03:46.660539+0000","last_became_active":"2024-09-18T11:45:25.381652+0000","last_became_peered":"2024-09-18T11:45:25.381652+0000","last_unstale":"2024-09-18T12:03:46.660539+0000","last_undegraded":"2024-09-18T12:03:46.660539+0000","last_fullsized":"2024-09-18T12:03:46.660539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:46.660227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:46.660227+0000","last_clean_scrub_stamp":"2024-09-18T12:03:46.660227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:18:33.805092+0000","scrub_duration":0.0038280499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:42.708151+0000","last_change":"2024-09-18T12:03:42.708095+0000","last_active":"2024-09-18T12:03:42.708151+0000","last_peered":"2024-09-18T12:03:42.708151+0000","last_clean":"2024-09-18T12:03:42.708151+0000","last_became_active":"2024-09-18T11:45:25.379657+0000","last_became_peered":"2024-09-18T11:45:25.379657+0000","last_unstale":"2024-09-18T12:03:42.708151+0000","last_undegraded":"2024-09-18T12:03:42.708151+0000","last_fullsized":"2024-09-18T12:03:42.708151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:42.707909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:42.707909+0000","last_clean_scrub_stamp":"2024-09-18T12:03:42.707909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:39:52.285391+0000","scrub_duration":0.0029633200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.631107+0000","last_change":"2024-09-18T12:03:45.631053+0000","last_active":"2024-09-18T12:03:45.631107+0000","last_peered":"2024-09-18T12:03:45.631107+0000","last_clean":"2024-09-18T12:03:45.631107+0000","last_became_active":"2024-09-18T11:45:25.338772+0000","last_became_peered":"2024-09-18T11:45:25.338772+0000","last_unstale":"2024-09-18T12:03:45.631107+0000","last_undegraded":"2024-09-18T12:03:45.631107+0000","last_fullsized":"2024-09-18T12:03:45.631107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:45.630856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:45.630856+0000","last_clean_scrub_stamp":"2024-09-18T12:03:45.630856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:25:04.540844+0000","scrub_duration":0.0028719840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:57.604068+0000","last_change":"2024-09-18T12:03:57.603971+0000","last_active":"2024-09-18T12:03:57.604068+0000","last_peered":"2024-09-18T12:03:57.604068+0000","last_clean":"2024-09-18T12:03:57.604068+0000","last_became_active":"2024-09-18T11:45:25.369294+0000","last_became_peered":"2024-09-18T11:45:25.369294+0000","last_unstale":"2024-09-18T12:03:57.604068+0000","last_undegraded":"2024-09-18T12:03:57.604068+0000","last_fullsized":"2024-09-18T12:03:57.604068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:57.603766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:57.603766+0000","last_clean_scrub_stamp":"2024-09-18T12:03:57.603766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:01:14.589358+0000","scrub_duration":0.0029912649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.799029+0000","last_change":"2024-09-18T12:03:51.798973+0000","last_active":"2024-09-18T12:03:51.799029+0000","last_peered":"2024-09-18T12:03:51.799029+0000","last_clean":"2024-09-18T12:03:51.799029+0000","last_became_active":"2024-09-18T11:45:25.393093+0000","last_became_peered":"2024-09-18T11:45:25.393093+0000","last_unstale":"2024-09-18T12:03:51.799029+0000","last_undegraded":"2024-09-18T12:03:51.799029+0000","last_fullsized":"2024-09-18T12:03:51.799029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:51.798768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:51.798768+0000","last_clean_scrub_stamp":"2024-09-18T12:03:51.798768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:53:20.875086+0000","scrub_duration":0.003211645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":648,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:02:11.911671+0000","last_change":"2024-09-18T11:45:25.382303+0000","last_active":"2024-09-18T12:02:11.911671+0000","last_peered":"2024-09-18T12:02:11.911671+0000","last_clean":"2024-09-18T12:02:11.911671+0000","last_became_active":"2024-09-18T11:45:25.380362+0000","last_became_peered":"2024-09-18T11:45:25.380362+0000","last_unstale":"2024-09-18T12:02:11.911671+0000","last_undegraded":"2024-09-18T12:02:11.911671+0000","last_fullsized":"2024-09-18T12:02:11.911671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:45:24.254207+0000","last_clean_scrub_stamp":"2024-09-18T11:45:24.254207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.578979+0000","last_change":"2024-09-18T12:03:51.578915+0000","last_active":"2024-09-18T12:03:51.578979+0000","last_peered":"2024-09-18T12:03:51.578979+0000","last_clean":"2024-09-18T12:03:51.578979+0000","last_became_active":"2024-09-18T11:45:25.357477+0000","last_became_peered":"2024-09-18T11:45:25.357477+0000","last_unstale":"2024-09-18T12:03:51.578979+0000","last_undegraded":"2024-09-18T12:03:51.578979+0000","last_fullsized":"2024-09-18T12:03:51.578979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:51.578715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:51.578715+0000","last_clean_scrub_stamp":"2024-09-18T12:03:51.578715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:26:29.948148+0000","scrub_duration":0.0031629409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":657,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:44.342218+0000","last_change":"2024-09-18T12:03:44.342106+0000","last_active":"2024-09-18T12:03:44.342218+0000","last_peered":"2024-09-18T12:03:44.342218+0000","last_clean":"2024-09-18T12:03:44.342218+0000","last_became_active":"2024-09-18T11:45:25.394579+0000","last_became_peered":"2024-09-18T11:45:25.394579+0000","last_unstale":"2024-09-18T12:03:44.342218+0000","last_undegraded":"2024-09-18T12:03:44.342218+0000","last_fullsized":"2024-09-18T12:03:44.342218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:44.341860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:44.341860+0000","last_clean_scrub_stamp":"2024-09-18T12:03:44.341860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:45:10.553793+0000","scrub_duration":0.00339669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:55.778734+0000","last_change":"2024-09-18T12:03:55.778657+0000","last_active":"2024-09-18T12:03:55.778734+0000","last_peered":"2024-09-18T12:03:55.778734+0000","last_clean":"2024-09-18T12:03:55.778734+0000","last_became_active":"2024-09-18T11:45:25.403069+0000","last_became_peered":"2024-09-18T11:45:25.403069+0000","last_unstale":"2024-09-18T12:03:55.778734+0000","last_undegraded":"2024-09-18T12:03:55.778734+0000","last_fullsized":"2024-09-18T12:03:55.778734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:55.778459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:55.778459+0000","last_clean_scrub_stamp":"2024-09-18T12:03:55.778459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:16:06.963815+0000","scrub_duration":0.0035369830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607793,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315060,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056780,"statfs":{"total":96636764160,"available":96314142720,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.88500000000000001,"15min":0.87},"min":{"1min":0.73799999999999999,"5min":0.48499999999999999,"15min":0.41499999999999998},"max":{"1min":1.1719999999999999,"5min":1.244,"15min":1.244},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.88100000000000001,"15min":0.85099999999999998},"min":{"1min":0.69899999999999995,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.099,"5min":1.294,"15min":1.294},"last":0.85999999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.83999999999999997,"15min":0.873},"min":{"1min":0.58699999999999997,"5min":0.52900000000000003,"15min":0.46500000000000002},"max":{"1min":1.0740000000000001,"5min":1.2529999999999999,"15min":1.325},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.86099999999999999,"15min":0.85999999999999999},"min":{"1min":0.70099999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.022,"5min":1.208,"15min":1.2689999999999999},"last":0.75}]},{"osd":2,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80200000000000005,"15min":0.81100000000000005},"min":{"1min":0.58499999999999996,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":1.1279999999999999,"5min":1.2410000000000001,"15min":1.423},"last":0.995},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.89800000000000002,"15min":0.86699999999999999},"min":{"1min":0.71099999999999997,"5min":0.503,"15min":0.439},"max":{"1min":1.2030000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.83299999999999996}]},{"osd":3,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.77400000000000002},"min":{"1min":0.56299999999999994,"5min":0.45800000000000002,"15min":0.38800000000000001},"max":{"1min":1.036,"5min":1.2689999999999999,"15min":1.6950000000000001},"last":0.627},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.752,"15min":0.77300000000000002},"min":{"1min":0.57899999999999996,"5min":0.439,"15min":0.373},"max":{"1min":1.0820000000000001,"5min":1.204,"15min":1.3580000000000001},"last":0.66900000000000004}]},{"osd":4,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.84799999999999998,"15min":0.86599999999999999},"min":{"1min":0.67500000000000004,"5min":0.376,"15min":0.376},"max":{"1min":1.155,"5min":1.159,"15min":1.323},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.96599999999999997,"15min":0.96299999999999997},"min":{"1min":0.60499999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.2609999999999999,"5min":1.3089999999999999,"15min":1.4670000000000001},"last":0.95499999999999996}]}]},{"osd":4,"up_from":12,"seq":51539607792,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319220,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052620,"statfs":{"total":96636764160,"available":96309882880,"internally_reserved":0,"allocated":9613312,"data_stored":4953859,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.83599999999999997,"15min":0.82799999999999996},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.47699999999999998},"max":{"1min":1.04,"5min":1.1870000000000001,"15min":1.3120000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.83099999999999996},"min":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.46400000000000002},"max":{"1min":0.96499999999999997,"5min":1.272,"15min":1.296},"last":0.69799999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.81399999999999995,"15min":0.83199999999999996},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.504},"max":{"1min":0.877,"5min":1.3169999999999999,"15min":1.331},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.89700000000000002,"15min":0.86299999999999999},"min":{"1min":0.61099999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0149999999999999,"5min":1.3380000000000001,"15min":1.3600000000000001},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79600000000000004,"15min":0.82499999999999996},"min":{"1min":0.70499999999999996,"5min":0.56799999999999995,"15min":0.41699999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.294},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.90400000000000003,"15min":0.90100000000000002},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.432},"max":{"1min":0.97999999999999998,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.871}]},{"osd":3,"last update":"Wed Sep 18 12:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.748,"15min":0.76000000000000001},"min":{"1min":0.54000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90500000000000003,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.623},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85699999999999998,"15min":0.89000000000000001},"min":{"1min":0.64600000000000002,"5min":0.5,"15min":0.44600000000000001},"max":{"1min":0.98999999999999999,"5min":1.4019999999999999,"15min":1.496},"last":0.79500000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.56999999999999995,"5min":0.53500000000000003,"15min":0.40100000000000002},"max":{"1min":1.085,"5min":1.4430000000000001,"15min":2.899},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0289999999999999,"15min":0.97199999999999998},"min":{"1min":0.76900000000000002,"5min":0.69299999999999995,"15min":0.56599999999999995},"max":{"1min":1.28,"5min":1.466,"15min":3.004},"last":0.95799999999999996}]}]},{"osd":3,"up_from":12,"seq":51539607792,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319276,"kb_used_data":9444,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052564,"statfs":{"total":96636764160,"available":96309825536,"internally_reserved":0,"allocated":9670656,"data_stored":5483791,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.74399999999999999,"15min":0.77700000000000002},"min":{"1min":0.59699999999999998,"5min":0.5,"15min":0.35999999999999999},"max":{"1min":0.92300000000000004,"5min":1.331,"15min":1.331},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.58499999999999996,"5min":0.56799999999999995,"15min":0.40999999999999998},"max":{"1min":1.135,"5min":1.2110000000000001,"15min":1.2470000000000001},"last":0.89400000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.88600000000000001,"15min":0.84899999999999998},"min":{"1min":0.66300000000000003,"5min":0.59699999999999998,"15min":0.505},"max":{"1min":0.96899999999999997,"5min":1.2829999999999999,"15min":1.4910000000000001},"last":0.872},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91900000000000004,"15min":0.90700000000000003},"min":{"1min":0.84399999999999997,"5min":0.55000000000000004,"15min":0.45800000000000002},"max":{"1min":1.03,"5min":1.325,"15min":1.4530000000000001},"last":0.92100000000000004}]},{"osd":2,"last update":"Wed Sep 18 12:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.78900000000000003,"15min":0.79100000000000004},"min":{"1min":0.626,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":0.92200000000000004,"5min":1.419,"15min":1.419},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91100000000000003,"15min":0.92000000000000004},"min":{"1min":0.65300000000000002,"5min":0.63600000000000001,"15min":0.441},"max":{"1min":1.0960000000000001,"5min":1.3280000000000001,"15min":1.357},"last":0.59699999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68200000000000005,"15min":0.71099999999999997},"min":{"1min":0.50800000000000001,"5min":0.49299999999999999,"15min":0.43099999999999999},"max":{"1min":1.073,"5min":1.2569999999999999,"15min":1.5009999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.98899999999999999,"15min":0.95799999999999996},"min":{"1min":0.70099999999999996,"5min":0.60099999999999998,"15min":0.53700000000000003},"max":{"1min":1.202,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.81699999999999995}]},{"osd":5,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.65500000000000003,"5min":0.54500000000000004,"15min":0.34699999999999998},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.802},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0629999999999999,"15min":1.069},"min":{"1min":0.91300000000000003,"5min":0.68600000000000005,"15min":0.63200000000000001},"max":{"1min":1.3340000000000001,"5min":1.581,"15min":1.9039999999999999},"last":0.93799999999999994}]}]},{"osd":2,"up_from":12,"seq":51539607792,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315060,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056780,"statfs":{"total":96636764160,"available":96314142720,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.33100000000000002},"max":{"1min":0.71699999999999997,"5min":1.212,"15min":1.212},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73099999999999998,"15min":0.72499999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.28699999999999998},"max":{"1min":0.85599999999999998,"5min":1.254,"15min":1.254},"last":0.85599999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.316},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.996},"last":0.503},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.69099999999999995,"15min":0.67600000000000005},"min":{"1min":0.54100000000000004,"5min":0.498,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":1.577,"15min":4.1699999999999999},"last":0.88500000000000001}]},{"osd":3,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.88500000000000001},"min":{"1min":0.63100000000000001,"5min":0.52500000000000002,"15min":0.36899999999999999},"max":{"1min":1.0229999999999999,"5min":1.3799999999999999,"15min":2.6389999999999998},"last":0.748},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86499999999999999,"15min":0.88500000000000001},"min":{"1min":0.66700000000000004,"5min":0.623,"15min":0.53400000000000003},"max":{"1min":1.069,"5min":1.2330000000000001,"15min":4.3310000000000004},"last":0.71899999999999997}]},{"osd":4,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.92500000000000004},"min":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.376},"max":{"1min":1.1579999999999999,"5min":1.2270000000000001,"15min":4.2270000000000003},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.997,"5min":0.98099999999999998,"15min":0.94899999999999995},"min":{"1min":0.89400000000000002,"5min":0.69599999999999995,"15min":0.55700000000000005},"max":{"1min":1.103,"5min":1.671,"15min":1.706},"last":1.04}]},{"osd":5,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.873,"15min":0.91500000000000004},"min":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.39000000000000001},"max":{"1min":1.0129999999999999,"5min":1.202,"15min":2.5960000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.93700000000000006,"15min":0.96499999999999997},"min":{"1min":0.82899999999999996,"5min":0.74299999999999999,"15min":0.65800000000000003},"max":{"1min":1.2330000000000001,"5min":1.4890000000000001,"15min":4.7590000000000003},"last":0.96899999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640495,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315112,"kb_used_data":9440,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056728,"statfs":{"total":96636764160,"available":96314089472,"internally_reserved":0,"allocated":9666560,"data_stored":5483786,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55400000000000005,"15min":0.57199999999999995},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.313},"max":{"1min":0.76800000000000002,"5min":0.82899999999999996,"15min":1.2170000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.64400000000000002,"15min":0.67400000000000004},"min":{"1min":0.47199999999999998,"5min":0.442,"15min":0.39900000000000002},"max":{"1min":0.77700000000000002,"5min":0.95899999999999996,"15min":1.0369999999999999},"last":0.77700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.63300000000000001,"15min":0.65200000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79600000000000004,"5min":0.94799999999999995,"15min":1.161},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.81200000000000006,"15min":0.77800000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.41799999999999998},"max":{"1min":0.90800000000000003,"5min":1.0660000000000001,"15min":1.1799999999999999},"last":0.66900000000000004}]},{"osd":3,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83999999999999997,"15min":0.85199999999999998},"min":{"1min":0.69799999999999995,"5min":0.58099999999999996,"15min":0.53700000000000003},"max":{"1min":1.1200000000000001,"5min":1.2250000000000001,"15min":1.349},"last":0.752},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.85899999999999999,"15min":0.89900000000000002},"min":{"1min":0.627,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.0069999999999999,"5min":1.157,"15min":1.486},"last":1.0069999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.83899999999999997,"15min":0.872},"min":{"1min":0.69799999999999995,"5min":0.58399999999999996,"15min":0.52000000000000002},"max":{"1min":1.0229999999999999,"5min":1.123,"15min":1.329},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.98799999999999999,"15min":0.94399999999999995},"min":{"1min":0.79700000000000004,"5min":0.71199999999999997,"15min":0.62},"max":{"1min":1.1379999999999999,"5min":1.262,"15min":1.4350000000000001},"last":0.93999999999999995}]},{"osd":5,"last update":"Wed Sep 18 12:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.90700000000000003,"15min":0.92400000000000004},"min":{"1min":0.79800000000000004,"5min":0.56100000000000005,"15min":0.504},"max":{"1min":1.0289999999999999,"5min":1.3300000000000001,"15min":2.5840000000000001},"last":0.875},{"interface":"front","average":{"1min":1.04,"5min":1.0209999999999999,"15min":0.97999999999999998},"min":{"1min":0.84799999999999998,"5min":0.67200000000000004,"15min":0.47399999999999998},"max":{"1min":1.2310000000000001,"5min":1.53,"15min":1.752},"last":0.82899999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640495,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315068,"kb_used_data":9396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056772,"statfs":{"total":96636764160,"available":96314134528,"internally_reserved":0,"allocated":9621504,"data_stored":4954882,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55400000000000005,"15min":0.57799999999999996},"min":{"1min":0.42099999999999999,"5min":0.34000000000000002,"15min":0.191},"max":{"1min":0.66600000000000004,"5min":2.4470000000000001,"15min":3.8849999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.73299999999999998,"15min":0.71299999999999997},"min":{"1min":0.49299999999999999,"5min":0.40300000000000002,"15min":0.26600000000000001},"max":{"1min":0.80500000000000005,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.23300000000000001},"max":{"1min":0.85399999999999998,"5min":1.3799999999999999,"15min":3.6030000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.86599999999999999,"15min":0.82299999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.30399999999999999},"max":{"1min":1.2749999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.745}]},{"osd":3,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85199999999999998,"15min":0.84899999999999998},"min":{"1min":0.65700000000000003,"5min":0.60099999999999998,"15min":0.28100000000000003},"max":{"1min":1.0589999999999999,"5min":2.2440000000000002,"15min":3.7909999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.93100000000000005,"15min":0.92200000000000004},"min":{"1min":0.71999999999999997,"5min":0.65800000000000003,"15min":0.313},"max":{"1min":1.0669999999999999,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.0429999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.028,"15min":0.96299999999999997},"min":{"1min":0.753,"5min":0.68600000000000005,"15min":0.40799999999999997},"max":{"1min":1.1950000000000001,"5min":2.8530000000000002,"15min":3.6709999999999998},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.022,"5min":0.99399999999999999,"15min":1},"min":{"1min":0.86499999999999999,"5min":0.67400000000000004,"15min":0.33700000000000002},"max":{"1min":1.165,"5min":2.726,"15min":3.5270000000000001},"last":1.165}]},{"osd":5,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.89400000000000002,"15min":0.88800000000000001},"min":{"1min":0.75900000000000001,"5min":0.66700000000000004,"15min":0.36799999999999999},"max":{"1min":1.0880000000000001,"5min":1.389,"15min":3.6469999999999998},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.98799999999999999,"15min":0.97199999999999998},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.39500000000000002},"max":{"1min":1.347,"5min":2.7429999999999999,"15min":4.3280000000000003},"last":0.97899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:04.155 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:04:04.156 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:04:04.156 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:04:04.156 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:04:04.156 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:04:04.156 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:04:04.157 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-18T11:45:24.254207+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=45, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=3, tm_sec=39, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:04:04.157 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T12:04:24.157 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:24.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient: get_monmap_and_config 2024-09-18T12:04:24.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient: build_initial_monmap 2024-09-18T12:04:24.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient: monmap: 2024-09-18T12:04:24.269 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:04:24.269 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.269 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:04:24.270228+0000 2024-09-18T12:04:24.269 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:04:24.270228+0000 2024-09-18T12:04:24.269 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:04:24.270 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.270 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:04:24.270 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:04:24.270 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:04:24.270 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 1 Processor -- start 2024-09-18T12:04:24.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 1 -- start start 2024-09-18T12:04:24.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient: init 2024-09-18T12:04:24.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient: _reopen_session rank -1 2024-09-18T12:04:24.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:04:24.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient(hunting): picked mon.noname-b con 0x7fbbb40a1230 addr v1:172.21.15.112:6790/0 2024-09-18T12:04:24.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient(hunting): picked mon.noname-c con 0x7fbbb406e9d0 addr v1:172.21.15.129:6789/0 2024-09-18T12:04:24.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient(hunting): picked mon.noname-a con 0x7fbbb4135300 addr v1:172.21.15.112:6789/0 2024-09-18T12:04:24.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb40a1600 con 0x7fbbb4135300 2024-09-18T12:04:24.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb406eda0 con 0x7fbbb406e9d0 2024-09-18T12:04:24.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb406b5a0 con 0x7fbbb40a1230 2024-09-18T12:04:24.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient(hunting): _renew_subs 2024-09-18T12:04:24.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbba177700 10 monclient(hunting): authenticate will time out at 2036.111529s 2024-09-18T12:04:24.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbb37fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fbbb40a1230 0x7fbbb406a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50118/0 (socket says 172.21.15.112:50118) 2024-09-18T12:04:24.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbb3fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbbb4135300 0x7fbbb41376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59676/0 (socket says 172.21.15.112:59676) 2024-09-18T12:04:24.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.268+0000 7fbbb37fe700 1 -- 172.21.15.112:0/1074127166 learned_addr learned my addr 172.21.15.112:0/1074127166 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:04:24.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb3fff700 1 -- 172.21.15.112:0/1074127166 >> v1:172.21.15.112:6789/0 conn(0x7fbbb4135300 legacy=0x7fbbb41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb37fe700 1 -- 172.21.15.112:0/1074127166 >> v1:172.21.15.112:6790/0 conn(0x7fbbb40a1230 legacy=0x7fbbb406a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb2ffd700 1 -- 172.21.15.112:0/1074127166 >> v1:172.21.15.129:6789/0 conn(0x7fbbb406e9d0 legacy=0x7fbbb406ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44579104 0 0) 0x7fbbb40a1600 con 0x7fbbb4135300 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): my global_id is 164244 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb9c003570 con 0x7fbbb4135300 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918136376 0 0) 0x7fbbb406b5a0 con 0x7fbbb40a1230 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): my global_id is 154340 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb40a1600 con 0x7fbbb40a1230 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539808752 0 0) 0x7fbbb406eda0 con 0x7fbbb406e9d0 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient(hunting): my global_id is 144427 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb406b5a0 con 0x7fbbb406e9d0 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704180026 0 0) 0x7fbbb40a1600 con 0x7fbbb40a1230 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 >> v1:172.21.15.129:6789/0 conn(0x7fbbb406e9d0 legacy=0x7fbbb406ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 >> v1:172.21.15.112:6789/0 conn(0x7fbbb4135300 legacy=0x7fbbb41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: _finish_hunting 0 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 monclient: found mon.noname-b 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.112:6790/0 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb409f9e0 con 0x7fbbb40a1230 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: _finish_auth 0 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 5 monclient: authenticate success, global_id 154340 2024-09-18T12:04:24.277 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbba8001920 con 0x7fbbb40a1230 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: dump: 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.278 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 <== mon.2 v1:172.21.15.112:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbba8001bb0 con 0x7fbbb40a1230 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 10 monclient: get_monmap_and_config success 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 4 set_mon_vals no callback set 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 1 -- 172.21.15.112:0/1074127166 <== mon.2 v1:172.21.15.112:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbba8001da0 con 0x7fbbb40a1230 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbb27fc700 10 monclient: dump: 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.279 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 10 monclient: shutdown 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 1 -- 172.21.15.112:0/1074127166 >> v1:172.21.15.112:6790/0 conn(0x7fbbb40a1230 legacy=0x7fbbb406a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 1 -- 172.21.15.112:0/1074127166 shutdown_connections 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.272+0000 7fbbba177700 1 -- 172.21.15.112:0/1074127166 >> 172.21.15.112:0/1074127166 conn(0x7fbbb40680d0 msgr2=0x7fbbb40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 -- 172.21.15.112:0/1074127166 shutdown_connections 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 -- 172.21.15.112:0/1074127166 wait complete. 2024-09-18T12:04:24.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient: build_initial_monmap 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient: monmap: 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:04:24.281 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 librados: starting msgr at 2024-09-18T12:04:24.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 librados: starting objecter 2024-09-18T12:04:24.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 Processor -- start 2024-09-18T12:04:24.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 -- start start 2024-09-18T12:04:24.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 librados: setting wanted keys 2024-09-18T12:04:24.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 librados: calling monclient init 2024-09-18T12:04:24.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient: init 2024-09-18T12:04:24.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient: _reopen_session rank -1 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient(hunting): picked mon.noname-c con 0x7fbbb406e9d0 addr v1:172.21.15.112:6790/0 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient(hunting): picked mon.noname-a con 0x7fbbb40a1230 addr v1:172.21.15.112:6789/0 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient(hunting): picked mon.noname-b con 0x7fbbb4135300 addr v1:172.21.15.129:6789/0 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb411f400 con 0x7fbbb40a1230 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb406ef90 con 0x7fbbb4135300 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb409cfe0 con 0x7fbbb406e9d0 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient(hunting): _renew_subs 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbba177700 10 monclient(hunting): authenticate will time out at 2036.118156s 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbb37fe700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fbbb406e9d0 0x7fbbb413d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50132/0 (socket says 172.21.15.112:50132) 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.276+0000 7fbbb37fe700 1 -- 172.21.15.112:0/1691819225 learned_addr learned my addr 172.21.15.112:0/1691819225 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbb37fe700 1 -- 172.21.15.112:0/1691819225 >> v1:172.21.15.112:6790/0 conn(0x7fbbb406e9d0 legacy=0x7fbbb413d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 client.?.objecter ms_handle_connect 0x7fbbb406e9d0 2024-09-18T12:04:24.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbb2ffd700 1 -- 172.21.15.112:0/1691819225 >> v1:172.21.15.112:6789/0 conn(0x7fbbb40a1230 legacy=0x7fbbb4142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 client.?.objecter ms_handle_connect 0x7fbbb40a1230 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125834955 0 0) 0x7fbbb409cfe0 con 0x7fbbb406e9d0 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): my global_id is 154343 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbb3fff700 1 -- 172.21.15.112:0/1691819225 >> v1:172.21.15.129:6789/0 conn(0x7fbbb4135300 legacy=0x7fbbb41460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba0004f10 con 0x7fbbb406e9d0 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 client.?.objecter ms_handle_connect 0x7fbbb4135300 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292937838 0 0) 0x7fbbb411f400 con 0x7fbbb40a1230 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): my global_id is 164247 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb409cfe0 con 0x7fbbb40a1230 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124710629 0 0) 0x7fbbb406ef90 con 0x7fbbb4135300 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): my global_id is 144430 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb411f400 con 0x7fbbb4135300 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114319178 0 0) 0x7fbba0004f10 con 0x7fbbb406e9d0 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbb406ef90 con 0x7fbbb406e9d0 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbba80049d0 con 0x7fbbb406e9d0 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): dump: 2024-09-18T12:04:24.289 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2945659160 0 0) 0x7fbbb409cfe0 con 0x7fbbb40a1230 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbba0004f10 con 0x7fbbb40a1230 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbba4001bc0 con 0x7fbbb40a1230 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient(hunting): dump: 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2484233565 0 0) 0x7fbba0004f10 con 0x7fbbb40a1230 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 >> v1:172.21.15.112:6790/0 conn(0x7fbbb406e9d0 legacy=0x7fbbb413d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 >> v1:172.21.15.129:6789/0 conn(0x7fbbb4135300 legacy=0x7fbbb41460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient: _finish_hunting 0 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 monclient: found mon.a 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb411cbc0 con 0x7fbbb40a1230 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient: _finish_auth 0 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 5 monclient: authenticate success, global_id 164247 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 10 monclient: _renew_subs 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.112:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb411c6b0 con 0x7fbbb40a1230 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 10 client.164247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 10 monclient: _renew_subs 2024-09-18T12:04:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.112:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb411d420 con 0x7fbbb40a1230 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 1 librados: init done 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbba40022b0 con 0x7fbbb40a1230 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fbba40025c0 con 0x7fbbb40a1230 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbb1ffb700 4 set_mon_vals no callback set 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 10 monclient: dump: 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.293 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.112:6789/0 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.112:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb940050b0 con 0x7fbbb40a1230 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.280+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fbba4002720 con 0x7fbbb40a1230 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.284+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fbba4043c50 con 0x7fbbb40a1230 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.284+0000 7fbb9b7fe700 10 client.164247.objecter ms_dispatch 0x7fbbb4000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.284+0000 7fbb9b7fe700 3 client.164247.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.284+0000 7fbb9b7fe700 3 client.164247.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:04:24.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.284+0000 7fbbb37fe700 10 monclient: get_auth_request con 0x7fbba00383a0 auth_method 0 2024-09-18T12:04:24.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.284+0000 7fbb9b7fe700 20 client.164247.objecter dump_active .. 0 homeless 2024-09-18T12:04:24.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.284+0000 7fbb9b7fe700 10 client.164247.objecter ms_handle_connect 0x7fbba00383a0 2024-09-18T12:04:24.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.292+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mon.0 v1:172.21.15.112:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fbba4002c60 con 0x7fbbb40a1230 2024-09-18T12:04:24.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.292+0000 7fbb9b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:24.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.292+0000 7fbb9b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:04:24.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.452+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 --> v1:172.21.15.129:6812/20806 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb94000b90 con 0x7fbba00383a0 2024-09-18T12:04:24.464 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.464+0000 7fbb9b7fe700 1 -- 172.21.15.112:0/1691819225 <== mgr.4104 v1:172.21.15.129:6812/20806 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134643 (unknown 2965378022 0 3741745161) 0x7fbb94000b90 con 0x7fbba00383a0 2024-09-18T12:04:24.466 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-18T12:04:24.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.468+0000 7fbbba177700 10 librados: watch_flush enter 2024-09-18T12:04:24.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.468+0000 7fbbba177700 10 librados: watch_flush exit 2024-09-18T12:04:24.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.468+0000 7fbbba177700 20 client.164247.objecter shutdown clearing up homeless session... 2024-09-18T12:04:24.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.468+0000 7fbbba177700 10 client.164247.objecter successfully canceled tick 2024-09-18T12:04:24.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.468+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 >> v1:172.21.15.129:6812/20806 conn(0x7fbba00383a0 legacy=0x7fbba003a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.468+0000 7fbbba177700 10 monclient: shutdown 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.468+0000 7fbbba177700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.468+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 >> v1:172.21.15.112:6789/0 conn(0x7fbbb40a1230 legacy=0x7fbbb4142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.472+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 shutdown_connections 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.472+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 >> 172.21.15.112:0/1691819225 conn(0x7fbbb40680d0 msgr2=0x7fbbb406d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.472+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 shutdown_connections 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.472+0000 7fbbba177700 1 -- 172.21.15.112:0/1691819225 wait complete. 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.472+0000 7fbbba177700 1 librados: shutdown 2024-09-18T12:04:24.472 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-18T12:04:24.485 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1297,"stamp":"2024-09-18T12:04:24.395937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1898796,"kb_used_data":56444,"kb_used_omap":9,"kb_used_meta":1842294,"kb_avail":564332244,"statfs":{"total":579820584960,"available":577876217856,"internally_reserved":0,"allocated":57798656,"data_stored":30786034,"data_compressed":320902,"data_compressed_allocated":876544,"data_compressed_original":2744320,"omap_allocated":10224,"internal_metadata":1886509072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003715"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":654,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:04:01.185804+0000","last_change":"2024-09-18T12:04:01.185745+0000","last_active":"2024-09-18T12:04:01.185804+0000","last_peered":"2024-09-18T12:04:01.185804+0000","last_clean":"2024-09-18T12:04:01.185804+0000","last_became_active":"2024-09-18T11:45:25.373642+0000","last_became_peered":"2024-09-18T11:45:25.373642+0000","last_unstale":"2024-09-18T12:04:01.185804+0000","last_undegraded":"2024-09-18T12:04:01.185804+0000","last_fullsized":"2024-09-18T12:04:01.185804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:04:01.185533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:04:01.185533+0000","last_clean_scrub_stamp":"2024-09-18T12:04:01.185533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:05:18.241237+0000","scrub_duration":0.0037938410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.015861+0000","last_change":"2024-09-18T12:03:58.015792+0000","last_active":"2024-09-18T12:03:58.015861+0000","last_peered":"2024-09-18T12:03:58.015861+0000","last_clean":"2024-09-18T12:03:58.015861+0000","last_became_active":"2024-09-18T11:45:25.328032+0000","last_became_peered":"2024-09-18T11:45:25.328032+0000","last_unstale":"2024-09-18T12:03:58.015861+0000","last_undegraded":"2024-09-18T12:03:58.015861+0000","last_fullsized":"2024-09-18T12:03:58.015861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:58.015589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:58.015589+0000","last_clean_scrub_stamp":"2024-09-18T12:03:58.015589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:56:03.382870+0000","scrub_duration":0.0031503450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.788584+0000","last_change":"2024-09-18T12:03:49.788515+0000","last_active":"2024-09-18T12:03:49.788584+0000","last_peered":"2024-09-18T12:03:49.788584+0000","last_clean":"2024-09-18T12:03:49.788584+0000","last_became_active":"2024-09-18T11:45:25.305872+0000","last_became_peered":"2024-09-18T11:45:25.305872+0000","last_unstale":"2024-09-18T12:03:49.788584+0000","last_undegraded":"2024-09-18T12:03:49.788584+0000","last_fullsized":"2024-09-18T12:03:49.788584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:49.788307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:49.788307+0000","last_clean_scrub_stamp":"2024-09-18T12:03:49.788307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:25:23.899899+0000","scrub_duration":0.0033892200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:56.980139+0000","last_change":"2024-09-18T12:03:56.980022+0000","last_active":"2024-09-18T12:03:56.980139+0000","last_peered":"2024-09-18T12:03:56.980139+0000","last_clean":"2024-09-18T12:03:56.980139+0000","last_became_active":"2024-09-18T11:45:25.411133+0000","last_became_peered":"2024-09-18T11:45:25.411133+0000","last_unstale":"2024-09-18T12:03:56.980139+0000","last_undegraded":"2024-09-18T12:03:56.980139+0000","last_fullsized":"2024-09-18T12:03:56.980139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:56.979808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:56.979808+0000","last_clean_scrub_stamp":"2024-09-18T12:03:56.979808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:08:59.245327+0000","scrub_duration":0.003648044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":654,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.184043+0000","last_change":"2024-09-18T12:03:53.183942+0000","last_active":"2024-09-18T12:03:53.184043+0000","last_peered":"2024-09-18T12:03:53.184043+0000","last_clean":"2024-09-18T12:03:53.184043+0000","last_became_active":"2024-09-18T11:45:25.399898+0000","last_became_peered":"2024-09-18T11:45:25.399898+0000","last_unstale":"2024-09-18T12:03:53.184043+0000","last_undegraded":"2024-09-18T12:03:53.184043+0000","last_fullsized":"2024-09-18T12:03:53.184043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:53.183699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:53.183699+0000","last_clean_scrub_stamp":"2024-09-18T12:03:53.183699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:03:01.050741+0000","scrub_duration":0.0035415669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:59.028630+0000","last_change":"2024-09-18T12:03:59.028574+0000","last_active":"2024-09-18T12:03:59.028630+0000","last_peered":"2024-09-18T12:03:59.028630+0000","last_clean":"2024-09-18T12:03:59.028630+0000","last_became_active":"2024-09-18T11:45:25.396975+0000","last_became_peered":"2024-09-18T11:45:25.396975+0000","last_unstale":"2024-09-18T12:03:59.028630+0000","last_undegraded":"2024-09-18T12:03:59.028630+0000","last_fullsized":"2024-09-18T12:03:59.028630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:59.028376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:59.028376+0000","last_clean_scrub_stamp":"2024-09-18T12:03:59.028376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:00:50.746697+0000","scrub_duration":0.0034269259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:04:11.713357+0000","last_change":"2024-09-18T12:04:11.713296+0000","last_active":"2024-09-18T12:04:11.713357+0000","last_peered":"2024-09-18T12:04:11.713357+0000","last_clean":"2024-09-18T12:04:11.713357+0000","last_became_active":"2024-09-18T11:45:25.340726+0000","last_became_peered":"2024-09-18T11:45:25.340726+0000","last_unstale":"2024-09-18T12:04:11.713357+0000","last_undegraded":"2024-09-18T12:04:11.713357+0000","last_fullsized":"2024-09-18T12:04:11.713357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:04:11.713074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:04:11.713074+0000","last_clean_scrub_stamp":"2024-09-18T12:04:11.713074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:31.035332+0000","scrub_duration":0.0032803530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":657,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:46.396477+0000","last_change":"2024-09-18T12:03:46.396412+0000","last_active":"2024-09-18T12:03:46.396477+0000","last_peered":"2024-09-18T12:03:46.396477+0000","last_clean":"2024-09-18T12:03:46.396477+0000","last_became_active":"2024-09-18T11:45:25.408464+0000","last_became_peered":"2024-09-18T11:45:25.408464+0000","last_unstale":"2024-09-18T12:03:46.396477+0000","last_undegraded":"2024-09-18T12:03:46.396477+0000","last_fullsized":"2024-09-18T12:03:46.396477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:46.396196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:46.396196+0000","last_clean_scrub_stamp":"2024-09-18T12:03:46.396196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:46:03.414527+0000","scrub_duration":0.0035477960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.670831+0000","last_change":"2024-09-18T12:03:58.670753+0000","last_active":"2024-09-18T12:03:58.670831+0000","last_peered":"2024-09-18T12:03:58.670831+0000","last_clean":"2024-09-18T12:03:58.670831+0000","last_became_active":"2024-09-18T11:45:25.343097+0000","last_became_peered":"2024-09-18T11:45:25.343097+0000","last_unstale":"2024-09-18T12:03:58.670831+0000","last_undegraded":"2024-09-18T12:03:58.670831+0000","last_fullsized":"2024-09-18T12:03:58.670831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:58.670528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:58.670528+0000","last_clean_scrub_stamp":"2024-09-18T12:03:58.670528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:29:07.343349+0000","scrub_duration":0.0032734999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:47.880890+0000","last_change":"2024-09-18T12:03:47.880826+0000","last_active":"2024-09-18T12:03:47.880890+0000","last_peered":"2024-09-18T12:03:47.880890+0000","last_clean":"2024-09-18T12:03:47.880890+0000","last_became_active":"2024-09-18T11:45:25.377194+0000","last_became_peered":"2024-09-18T11:45:25.377194+0000","last_unstale":"2024-09-18T12:03:47.880890+0000","last_undegraded":"2024-09-18T12:03:47.880890+0000","last_fullsized":"2024-09-18T12:03:47.880890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:47.880598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:47.880598+0000","last_clean_scrub_stamp":"2024-09-18T12:03:47.880598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:53:32.792745+0000","scrub_duration":0.0034949949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":654,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:04:03.217737+0000","last_change":"2024-09-18T12:04:03.217666+0000","last_active":"2024-09-18T12:04:03.217737+0000","last_peered":"2024-09-18T12:04:03.217737+0000","last_clean":"2024-09-18T12:04:03.217737+0000","last_became_active":"2024-09-18T11:45:25.341682+0000","last_became_peered":"2024-09-18T11:45:25.341682+0000","last_unstale":"2024-09-18T12:04:03.217737+0000","last_undegraded":"2024-09-18T12:04:03.217737+0000","last_fullsized":"2024-09-18T12:04:03.217737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:04:03.217446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:04:03.217446+0000","last_clean_scrub_stamp":"2024-09-18T12:04:03.217446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:30:36.074927+0000","scrub_duration":0.003191677,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":654,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:04:07.191611+0000","last_change":"2024-09-18T12:04:07.191537+0000","last_active":"2024-09-18T12:04:07.191611+0000","last_peered":"2024-09-18T12:04:07.191611+0000","last_clean":"2024-09-18T12:04:07.191611+0000","last_became_active":"2024-09-18T11:45:25.377663+0000","last_became_peered":"2024-09-18T11:45:25.377663+0000","last_unstale":"2024-09-18T12:04:07.191611+0000","last_undegraded":"2024-09-18T12:04:07.191611+0000","last_fullsized":"2024-09-18T12:04:07.191611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:04:07.191316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:04:07.191316+0000","last_clean_scrub_stamp":"2024-09-18T12:04:07.191316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:46:49.144279+0000","scrub_duration":0.0036620810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":660,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:46.802350+0000","last_change":"2024-09-18T12:03:46.802290+0000","last_active":"2024-09-18T12:03:46.802350+0000","last_peered":"2024-09-18T12:03:46.802350+0000","last_clean":"2024-09-18T12:03:46.802350+0000","last_became_active":"2024-09-18T11:44:21.098366+0000","last_became_peered":"2024-09-18T11:44:21.098366+0000","last_unstale":"2024-09-18T12:03:46.802350+0000","last_undegraded":"2024-09-18T12:03:46.802350+0000","last_fullsized":"2024-09-18T12:03:46.802350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:46.802087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:46.802087+0000","last_clean_scrub_stamp":"2024-09-18T12:03:46.802087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:05:34.411795+0000","scrub_duration":0.003807094,"objects_trimmed":0,"snaptrim_duration":0.00068350500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":657,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.304300+0000","last_change":"2024-09-18T12:03:49.304241+0000","last_active":"2024-09-18T12:03:49.304300+0000","last_peered":"2024-09-18T12:03:49.304300+0000","last_clean":"2024-09-18T12:03:49.304300+0000","last_became_active":"2024-09-18T11:45:25.405515+0000","last_became_peered":"2024-09-18T11:45:25.405515+0000","last_unstale":"2024-09-18T12:03:49.304300+0000","last_undegraded":"2024-09-18T12:03:49.304300+0000","last_fullsized":"2024-09-18T12:03:49.304300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:49.304048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:49.304048+0000","last_clean_scrub_stamp":"2024-09-18T12:03:49.304048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:51:38.831116+0000","scrub_duration":0.0034724690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":669,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.640895+0000","last_change":"2024-09-18T12:03:45.640825+0000","last_active":"2024-09-18T12:03:45.640895+0000","last_peered":"2024-09-18T12:03:45.640895+0000","last_clean":"2024-09-18T12:03:45.640895+0000","last_became_active":"2024-09-18T11:44:21.273425+0000","last_became_peered":"2024-09-18T11:44:21.273425+0000","last_unstale":"2024-09-18T12:03:45.640895+0000","last_undegraded":"2024-09-18T12:03:45.640895+0000","last_fullsized":"2024-09-18T12:03:45.640895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:45.640579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:45.640579+0000","last_clean_scrub_stamp":"2024-09-18T12:03:45.640579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:27:16.770756+0000","scrub_duration":0.0037749910000000001,"objects_trimmed":0,"snaptrim_duration":0.00066149999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.825739+0000","last_change":"2024-09-18T12:03:53.825666+0000","last_active":"2024-09-18T12:03:53.825739+0000","last_peered":"2024-09-18T12:03:53.825739+0000","last_clean":"2024-09-18T12:03:53.825739+0000","last_became_active":"2024-09-18T11:45:25.305686+0000","last_became_peered":"2024-09-18T11:45:25.305686+0000","last_unstale":"2024-09-18T12:03:53.825739+0000","last_undegraded":"2024-09-18T12:03:53.825739+0000","last_fullsized":"2024-09-18T12:03:53.825739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:53.825473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:53.825473+0000","last_clean_scrub_stamp":"2024-09-18T12:03:53.825473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:07:07.289345+0000","scrub_duration":0.0035215210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":669,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.849844+0000","last_change":"2024-09-18T12:03:45.849749+0000","last_active":"2024-09-18T12:03:45.849844+0000","last_peered":"2024-09-18T12:03:45.849844+0000","last_clean":"2024-09-18T12:03:45.849844+0000","last_became_active":"2024-09-18T11:44:22.201555+0000","last_became_peered":"2024-09-18T11:44:22.201555+0000","last_unstale":"2024-09-18T12:03:45.849844+0000","last_undegraded":"2024-09-18T12:03:45.849844+0000","last_fullsized":"2024-09-18T12:03:45.849844+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:45.849511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:45.849511+0000","last_clean_scrub_stamp":"2024-09-18T12:03:45.849511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:48.324946+0000","scrub_duration":0.0035045440000000001,"objects_trimmed":0,"snaptrim_duration":0.00083892500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"699'8","reported_seq":672,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:54.702984+0000","last_change":"2024-09-18T12:03:54.702924+0000","last_active":"2024-09-18T12:03:54.702984+0000","last_peered":"2024-09-18T12:03:54.702984+0000","last_clean":"2024-09-18T12:03:54.702984+0000","last_became_active":"2024-09-18T11:45:25.403586+0000","last_became_peered":"2024-09-18T11:45:25.403586+0000","last_unstale":"2024-09-18T12:03:54.702984+0000","last_undegraded":"2024-09-18T12:03:54.702984+0000","last_fullsized":"2024-09-18T12:03:54.702984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'8","last_scrub_stamp":"2024-09-18T12:03:54.702747+0000","last_deep_scrub":"699'8","last_deep_scrub_stamp":"2024-09-18T12:03:54.702747+0000","last_clean_scrub_stamp":"2024-09-18T12:03:54.702747+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:25:10.453350+0000","scrub_duration":0.0064599619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":665,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:47.165243+0000","last_change":"2024-09-18T12:03:47.165176+0000","last_active":"2024-09-18T12:03:47.165243+0000","last_peered":"2024-09-18T12:03:47.165243+0000","last_clean":"2024-09-18T12:03:47.165243+0000","last_became_active":"2024-09-18T11:44:21.099323+0000","last_became_peered":"2024-09-18T11:44:21.099323+0000","last_unstale":"2024-09-18T12:03:47.165243+0000","last_undegraded":"2024-09-18T12:03:47.165243+0000","last_fullsized":"2024-09-18T12:03:47.165243+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:47.164959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:47.164959+0000","last_clean_scrub_stamp":"2024-09-18T12:03:47.164959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:10:06.246346+0000","scrub_duration":0.0033963439999999999,"objects_trimmed":0,"snaptrim_duration":0.00043864400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:55.532841+0000","last_change":"2024-09-18T12:03:55.532787+0000","last_active":"2024-09-18T12:03:55.532841+0000","last_peered":"2024-09-18T12:03:55.532841+0000","last_clean":"2024-09-18T12:03:55.532841+0000","last_became_active":"2024-09-18T11:45:25.367362+0000","last_became_peered":"2024-09-18T11:45:25.367362+0000","last_unstale":"2024-09-18T12:03:55.532841+0000","last_undegraded":"2024-09-18T12:03:55.532841+0000","last_fullsized":"2024-09-18T12:03:55.532841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:55.532635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:55.532635+0000","last_clean_scrub_stamp":"2024-09-18T12:03:55.532635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:45:02.887722+0000","scrub_duration":0.0026179189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"699'74","reported_seq":756,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.712591+0000","last_change":"2024-09-18T12:03:53.712497+0000","last_active":"2024-09-18T12:03:53.712591+0000","last_peered":"2024-09-18T12:03:53.712591+0000","last_clean":"2024-09-18T12:03:53.712591+0000","last_became_active":"2024-09-18T11:44:21.097521+0000","last_became_peered":"2024-09-18T11:44:21.097521+0000","last_unstale":"2024-09-18T12:03:53.712591+0000","last_undegraded":"2024-09-18T12:03:53.712591+0000","last_fullsized":"2024-09-18T12:03:53.712591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'74","last_scrub_stamp":"2024-09-18T12:03:53.712046+0000","last_deep_scrub":"699'74","last_deep_scrub_stamp":"2024-09-18T12:03:53.712046+0000","last_clean_scrub_stamp":"2024-09-18T12:03:53.712046+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:50.861870+0000","scrub_duration":0.0099226230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"699'2","reported_seq":667,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.137599+0000","last_change":"2024-09-18T12:03:45.137534+0000","last_active":"2024-09-18T12:03:45.137599+0000","last_peered":"2024-09-18T12:03:45.137599+0000","last_clean":"2024-09-18T12:03:45.137599+0000","last_became_active":"2024-09-18T11:44:21.099921+0000","last_became_peered":"2024-09-18T11:44:21.099921+0000","last_unstale":"2024-09-18T12:03:45.137599+0000","last_undegraded":"2024-09-18T12:03:45.137599+0000","last_fullsized":"2024-09-18T12:03:45.137599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'2","last_scrub_stamp":"2024-09-18T12:03:45.137311+0000","last_deep_scrub":"699'2","last_deep_scrub_stamp":"2024-09-18T12:03:45.137311+0000","last_clean_scrub_stamp":"2024-09-18T12:03:45.137311+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:20:51.917782+0000","scrub_duration":0.006619219,"objects_trimmed":0,"snaptrim_duration":0.00063767800000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:56.760649+0000","last_change":"2024-09-18T12:03:56.760584+0000","last_active":"2024-09-18T12:03:56.760649+0000","last_peered":"2024-09-18T12:03:56.760649+0000","last_clean":"2024-09-18T12:03:56.760649+0000","last_became_active":"2024-09-18T11:45:25.406430+0000","last_became_peered":"2024-09-18T11:45:25.406430+0000","last_unstale":"2024-09-18T12:03:56.760649+0000","last_undegraded":"2024-09-18T12:03:56.760649+0000","last_fullsized":"2024-09-18T12:03:56.760649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:56.760370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:56.760370+0000","last_clean_scrub_stamp":"2024-09-18T12:03:56.760370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:49:03.599476+0000","scrub_duration":0.0033169749999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"699'3","reported_seq":672,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:43.797405+0000","last_change":"2024-09-18T12:03:43.797302+0000","last_active":"2024-09-18T12:03:43.797405+0000","last_peered":"2024-09-18T12:03:43.797405+0000","last_clean":"2024-09-18T12:03:43.797405+0000","last_became_active":"2024-09-18T11:44:22.812905+0000","last_became_peered":"2024-09-18T11:44:22.812905+0000","last_unstale":"2024-09-18T12:03:43.797405+0000","last_undegraded":"2024-09-18T12:03:43.797405+0000","last_fullsized":"2024-09-18T12:03:43.797405+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'3","last_scrub_stamp":"2024-09-18T12:03:43.796989+0000","last_deep_scrub":"699'3","last_deep_scrub_stamp":"2024-09-18T12:03:43.796989+0000","last_clean_scrub_stamp":"2024-09-18T12:03:43.796989+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:48:51.634241+0000","scrub_duration":0.0066051479999999999,"objects_trimmed":0,"snaptrim_duration":0.0021163060000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.607421+0000","last_change":"2024-09-18T12:03:49.607335+0000","last_active":"2024-09-18T12:03:49.607421+0000","last_peered":"2024-09-18T12:03:49.607421+0000","last_clean":"2024-09-18T12:03:49.607421+0000","last_became_active":"2024-09-18T11:45:25.363235+0000","last_became_peered":"2024-09-18T11:45:25.363235+0000","last_unstale":"2024-09-18T12:03:49.607421+0000","last_undegraded":"2024-09-18T12:03:49.607421+0000","last_fullsized":"2024-09-18T12:03:49.607421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:49.607001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:49.607001+0000","last_clean_scrub_stamp":"2024-09-18T12:03:49.607001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:18:26.489548+0000","scrub_duration":0.003314993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":660,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:47.787279+0000","last_change":"2024-09-18T12:03:47.787216+0000","last_active":"2024-09-18T12:03:47.787279+0000","last_peered":"2024-09-18T12:03:47.787279+0000","last_clean":"2024-09-18T12:03:47.787279+0000","last_became_active":"2024-09-18T11:44:21.274053+0000","last_became_peered":"2024-09-18T11:44:21.274053+0000","last_unstale":"2024-09-18T12:03:47.787279+0000","last_undegraded":"2024-09-18T12:03:47.787279+0000","last_fullsized":"2024-09-18T12:03:47.787279+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:47.787025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:47.787025+0000","last_clean_scrub_stamp":"2024-09-18T12:03:47.787025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:26:47.388077+0000","scrub_duration":0.0031932520000000002,"objects_trimmed":0,"snaptrim_duration":0.00065469200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":657,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:47.372331+0000","last_change":"2024-09-18T12:03:47.372270+0000","last_active":"2024-09-18T12:03:47.372331+0000","last_peered":"2024-09-18T12:03:47.372331+0000","last_clean":"2024-09-18T12:03:47.372331+0000","last_became_active":"2024-09-18T11:45:25.356178+0000","last_became_peered":"2024-09-18T11:45:25.356178+0000","last_unstale":"2024-09-18T12:03:47.372331+0000","last_undegraded":"2024-09-18T12:03:47.372331+0000","last_fullsized":"2024-09-18T12:03:47.372331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:47.372089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:47.372089+0000","last_clean_scrub_stamp":"2024-09-18T12:03:47.372089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:45:31.859250+0000","scrub_duration":0.0033298849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":669,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:42.617427+0000","last_change":"2024-09-18T12:03:42.617368+0000","last_active":"2024-09-18T12:03:42.617427+0000","last_peered":"2024-09-18T12:03:42.617427+0000","last_clean":"2024-09-18T12:03:42.617427+0000","last_became_active":"2024-09-18T11:44:21.273251+0000","last_became_peered":"2024-09-18T11:44:21.273251+0000","last_unstale":"2024-09-18T12:03:42.617427+0000","last_undegraded":"2024-09-18T12:03:42.617427+0000","last_fullsized":"2024-09-18T12:03:42.617427+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:42.617180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:42.617180+0000","last_clean_scrub_stamp":"2024-09-18T12:03:42.617180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:30:43.297276+0000","scrub_duration":0.0031515900000000001,"objects_trimmed":0,"snaptrim_duration":0.00058423399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.927805+0000","last_change":"2024-09-18T12:03:51.927734+0000","last_active":"2024-09-18T12:03:51.927805+0000","last_peered":"2024-09-18T12:03:51.927805+0000","last_clean":"2024-09-18T12:03:51.927805+0000","last_became_active":"2024-09-18T11:45:25.396428+0000","last_became_peered":"2024-09-18T11:45:25.396428+0000","last_unstale":"2024-09-18T12:03:51.927805+0000","last_undegraded":"2024-09-18T12:03:51.927805+0000","last_fullsized":"2024-09-18T12:03:51.927805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:51.927549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:51.927549+0000","last_clean_scrub_stamp":"2024-09-18T12:03:51.927549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:05:50.993805+0000","scrub_duration":0.003637361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:04:09.735843+0000","last_change":"2024-09-18T12:04:09.735781+0000","last_active":"2024-09-18T12:04:09.735843+0000","last_peered":"2024-09-18T12:04:09.735843+0000","last_clean":"2024-09-18T12:04:09.735843+0000","last_became_active":"2024-09-18T11:45:25.399695+0000","last_became_peered":"2024-09-18T11:45:25.399695+0000","last_unstale":"2024-09-18T12:04:09.735843+0000","last_undegraded":"2024-09-18T12:04:09.735843+0000","last_fullsized":"2024-09-18T12:04:09.735843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:04:09.735573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:04:09.735573+0000","last_clean_scrub_stamp":"2024-09-18T12:04:09.735573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:49:00.781706+0000","scrub_duration":0.003583096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:57.757282+0000","last_change":"2024-09-18T12:03:57.757190+0000","last_active":"2024-09-18T12:03:57.757282+0000","last_peered":"2024-09-18T12:03:57.757282+0000","last_clean":"2024-09-18T12:03:57.757282+0000","last_became_active":"2024-09-18T11:45:25.386932+0000","last_became_peered":"2024-09-18T11:45:25.386932+0000","last_unstale":"2024-09-18T12:03:57.757282+0000","last_undegraded":"2024-09-18T12:03:57.757282+0000","last_fullsized":"2024-09-18T12:03:57.757282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:57.756872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:57.756872+0000","last_clean_scrub_stamp":"2024-09-18T12:03:57.756872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:43:49.029283+0000","scrub_duration":0.0037374650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":654,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:04:04.237277+0000","last_change":"2024-09-18T12:04:04.237224+0000","last_active":"2024-09-18T12:04:04.237277+0000","last_peered":"2024-09-18T12:04:04.237277+0000","last_clean":"2024-09-18T12:04:04.237277+0000","last_became_active":"2024-09-18T11:45:25.401979+0000","last_became_peered":"2024-09-18T11:45:25.401979+0000","last_unstale":"2024-09-18T12:04:04.237277+0000","last_undegraded":"2024-09-18T12:04:04.237277+0000","last_fullsized":"2024-09-18T12:04:04.237277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:04:04.237028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:04:04.237028+0000","last_clean_scrub_stamp":"2024-09-18T12:04:04.237028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:20:17.212021+0000","scrub_duration":0.0034180949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:46.660539+0000","last_change":"2024-09-18T12:03:46.660473+0000","last_active":"2024-09-18T12:03:46.660539+0000","last_peered":"2024-09-18T12:03:46.660539+0000","last_clean":"2024-09-18T12:03:46.660539+0000","last_became_active":"2024-09-18T11:45:25.381652+0000","last_became_peered":"2024-09-18T11:45:25.381652+0000","last_unstale":"2024-09-18T12:03:46.660539+0000","last_undegraded":"2024-09-18T12:03:46.660539+0000","last_fullsized":"2024-09-18T12:03:46.660539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:46.660227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:46.660227+0000","last_clean_scrub_stamp":"2024-09-18T12:03:46.660227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:18:33.805092+0000","scrub_duration":0.0038280499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:42.708151+0000","last_change":"2024-09-18T12:03:42.708095+0000","last_active":"2024-09-18T12:03:42.708151+0000","last_peered":"2024-09-18T12:03:42.708151+0000","last_clean":"2024-09-18T12:03:42.708151+0000","last_became_active":"2024-09-18T11:45:25.379657+0000","last_became_peered":"2024-09-18T11:45:25.379657+0000","last_unstale":"2024-09-18T12:03:42.708151+0000","last_undegraded":"2024-09-18T12:03:42.708151+0000","last_fullsized":"2024-09-18T12:03:42.708151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:42.707909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:42.707909+0000","last_clean_scrub_stamp":"2024-09-18T12:03:42.707909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:39:52.285391+0000","scrub_duration":0.0029633200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.631107+0000","last_change":"2024-09-18T12:03:45.631053+0000","last_active":"2024-09-18T12:03:45.631107+0000","last_peered":"2024-09-18T12:03:45.631107+0000","last_clean":"2024-09-18T12:03:45.631107+0000","last_became_active":"2024-09-18T11:45:25.338772+0000","last_became_peered":"2024-09-18T11:45:25.338772+0000","last_unstale":"2024-09-18T12:03:45.631107+0000","last_undegraded":"2024-09-18T12:03:45.631107+0000","last_fullsized":"2024-09-18T12:03:45.631107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:45.630856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:45.630856+0000","last_clean_scrub_stamp":"2024-09-18T12:03:45.630856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:25:04.540844+0000","scrub_duration":0.0028719840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:57.604068+0000","last_change":"2024-09-18T12:03:57.603971+0000","last_active":"2024-09-18T12:03:57.604068+0000","last_peered":"2024-09-18T12:03:57.604068+0000","last_clean":"2024-09-18T12:03:57.604068+0000","last_became_active":"2024-09-18T11:45:25.369294+0000","last_became_peered":"2024-09-18T11:45:25.369294+0000","last_unstale":"2024-09-18T12:03:57.604068+0000","last_undegraded":"2024-09-18T12:03:57.604068+0000","last_fullsized":"2024-09-18T12:03:57.604068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:57.603766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:57.603766+0000","last_clean_scrub_stamp":"2024-09-18T12:03:57.603766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:01:14.589358+0000","scrub_duration":0.0029912649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.799029+0000","last_change":"2024-09-18T12:03:51.798973+0000","last_active":"2024-09-18T12:03:51.799029+0000","last_peered":"2024-09-18T12:03:51.799029+0000","last_clean":"2024-09-18T12:03:51.799029+0000","last_became_active":"2024-09-18T11:45:25.393093+0000","last_became_peered":"2024-09-18T11:45:25.393093+0000","last_unstale":"2024-09-18T12:03:51.799029+0000","last_undegraded":"2024-09-18T12:03:51.799029+0000","last_fullsized":"2024-09-18T12:03:51.799029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:51.798768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:51.798768+0000","last_clean_scrub_stamp":"2024-09-18T12:03:51.798768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:53:20.875086+0000","scrub_duration":0.003211645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":658,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:04:13.726293+0000","last_change":"2024-09-18T12:04:13.726190+0000","last_active":"2024-09-18T12:04:13.726293+0000","last_peered":"2024-09-18T12:04:13.726293+0000","last_clean":"2024-09-18T12:04:13.726293+0000","last_became_active":"2024-09-18T11:45:25.380362+0000","last_became_peered":"2024-09-18T11:45:25.380362+0000","last_unstale":"2024-09-18T12:04:13.726293+0000","last_undegraded":"2024-09-18T12:04:13.726293+0000","last_fullsized":"2024-09-18T12:04:13.726293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:04:13.725977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:04:13.725977+0000","last_clean_scrub_stamp":"2024-09-18T12:04:13.725977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:15:11.616076+0000","scrub_duration":0.0036677049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":656,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.578979+0000","last_change":"2024-09-18T12:03:51.578915+0000","last_active":"2024-09-18T12:03:51.578979+0000","last_peered":"2024-09-18T12:03:51.578979+0000","last_clean":"2024-09-18T12:03:51.578979+0000","last_became_active":"2024-09-18T11:45:25.357477+0000","last_became_peered":"2024-09-18T11:45:25.357477+0000","last_unstale":"2024-09-18T12:03:51.578979+0000","last_undegraded":"2024-09-18T12:03:51.578979+0000","last_fullsized":"2024-09-18T12:03:51.578979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:51.578715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:51.578715+0000","last_clean_scrub_stamp":"2024-09-18T12:03:51.578715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:26:29.948148+0000","scrub_duration":0.0031629409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":657,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:44.342218+0000","last_change":"2024-09-18T12:03:44.342106+0000","last_active":"2024-09-18T12:03:44.342218+0000","last_peered":"2024-09-18T12:03:44.342218+0000","last_clean":"2024-09-18T12:03:44.342218+0000","last_became_active":"2024-09-18T11:45:25.394579+0000","last_became_peered":"2024-09-18T11:45:25.394579+0000","last_unstale":"2024-09-18T12:03:44.342218+0000","last_undegraded":"2024-09-18T12:03:44.342218+0000","last_fullsized":"2024-09-18T12:03:44.342218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:44.341860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:44.341860+0000","last_clean_scrub_stamp":"2024-09-18T12:03:44.341860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:45:10.553793+0000","scrub_duration":0.00339669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-18T12:03:55.778734+0000","last_change":"2024-09-18T12:03:55.778657+0000","last_active":"2024-09-18T12:03:55.778734+0000","last_peered":"2024-09-18T12:03:55.778734+0000","last_clean":"2024-09-18T12:03:55.778734+0000","last_became_active":"2024-09-18T11:45:25.403069+0000","last_became_peered":"2024-09-18T11:45:25.403069+0000","last_unstale":"2024-09-18T12:03:55.778734+0000","last_undegraded":"2024-09-18T12:03:55.778734+0000","last_fullsized":"2024-09-18T12:03:55.778734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:03:55.778459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:03:55.778459+0000","last_clean_scrub_stamp":"2024-09-18T12:03:55.778459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:16:06.963815+0000","scrub_duration":0.0035369830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10714,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607797,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315060,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056780,"statfs":{"total":96636764160,"available":96314142720,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.88500000000000001,"15min":0.87},"min":{"1min":0.73799999999999999,"5min":0.48499999999999999,"15min":0.41499999999999998},"max":{"1min":1.1719999999999999,"5min":1.244,"15min":1.244},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.88100000000000001,"15min":0.85099999999999998},"min":{"1min":0.69899999999999995,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.099,"5min":1.294,"15min":1.294},"last":1.004}]},{"osd":1,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.83999999999999997,"15min":0.873},"min":{"1min":0.58699999999999997,"5min":0.52900000000000003,"15min":0.46500000000000002},"max":{"1min":1.0740000000000001,"5min":1.2529999999999999,"15min":1.325},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.86099999999999999,"15min":0.85999999999999999},"min":{"1min":0.70099999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.022,"5min":1.208,"15min":1.2689999999999999},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80200000000000005,"15min":0.81100000000000005},"min":{"1min":0.58499999999999996,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":1.1279999999999999,"5min":1.2410000000000001,"15min":1.423},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.89800000000000002,"15min":0.86699999999999999},"min":{"1min":0.71099999999999997,"5min":0.503,"15min":0.439},"max":{"1min":1.2030000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.81499999999999995}]},{"osd":3,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.77400000000000002},"min":{"1min":0.56299999999999994,"5min":0.45800000000000002,"15min":0.38800000000000001},"max":{"1min":1.036,"5min":1.2689999999999999,"15min":1.6950000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.752,"15min":0.77300000000000002},"min":{"1min":0.57899999999999996,"5min":0.439,"15min":0.373},"max":{"1min":1.0820000000000001,"5min":1.204,"15min":1.3580000000000001},"last":0.91300000000000003}]},{"osd":4,"last update":"Wed Sep 18 12:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.84799999999999998,"15min":0.86599999999999999},"min":{"1min":0.67500000000000004,"5min":0.376,"15min":0.376},"max":{"1min":1.155,"5min":1.159,"15min":1.323},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.96599999999999997,"15min":0.96299999999999997},"min":{"1min":0.60499999999999998,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.2609999999999999,"5min":1.3089999999999999,"15min":1.4670000000000001},"last":1.095}]}]},{"osd":4,"up_from":12,"seq":51539607796,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319220,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052620,"statfs":{"total":96636764160,"available":96309882880,"internally_reserved":0,"allocated":9613312,"data_stored":4953859,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.83599999999999997,"15min":0.82799999999999996},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.47699999999999998},"max":{"1min":1.04,"5min":1.1870000000000001,"15min":1.3120000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.83099999999999996},"min":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.46400000000000002},"max":{"1min":0.96499999999999997,"5min":1.272,"15min":1.296},"last":0.79000000000000004}]},{"osd":1,"last update":"Wed Sep 18 12:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.81399999999999995,"15min":0.83199999999999996},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.504},"max":{"1min":0.877,"5min":1.3169999999999999,"15min":1.331},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.89700000000000002,"15min":0.86299999999999999},"min":{"1min":0.61099999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0149999999999999,"5min":1.3380000000000001,"15min":1.3600000000000001},"last":0.84799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79600000000000004,"15min":0.82499999999999996},"min":{"1min":0.70499999999999996,"5min":0.56799999999999995,"15min":0.41699999999999998},"max":{"1min":1.125,"5min":1.125,"15min":1.294},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.90400000000000003,"15min":0.90100000000000002},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.432},"max":{"1min":0.97999999999999998,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.876}]},{"osd":3,"last update":"Wed Sep 18 12:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.748,"15min":0.76000000000000001},"min":{"1min":0.54000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90500000000000003,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85699999999999998,"15min":0.89000000000000001},"min":{"1min":0.64600000000000002,"5min":0.5,"15min":0.44600000000000001},"max":{"1min":0.98999999999999999,"5min":1.4019999999999999,"15min":1.496},"last":0.76600000000000001}]},{"osd":5,"last update":"Wed Sep 18 12:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.79000000000000004,"15min":0.78700000000000003},"min":{"1min":0.55100000000000005,"5min":0.53500000000000003,"15min":0.40100000000000002},"max":{"1min":1.0269999999999999,"5min":1.4430000000000001,"15min":2.899},"last":0.623},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.004,"15min":0.97699999999999998},"min":{"1min":0.77300000000000002,"5min":0.69299999999999995,"15min":0.56599999999999995},"max":{"1min":1.044,"5min":1.466,"15min":3.004},"last":0.82399999999999995}]}]},{"osd":3,"up_from":12,"seq":51539607796,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319276,"kb_used_data":9444,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052564,"statfs":{"total":96636764160,"available":96309825536,"internally_reserved":0,"allocated":9670656,"data_stored":5483791,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.74399999999999999,"15min":0.77700000000000002},"min":{"1min":0.59699999999999998,"5min":0.5,"15min":0.35999999999999999},"max":{"1min":0.92300000000000004,"5min":1.331,"15min":1.331},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.58499999999999996,"5min":0.56799999999999995,"15min":0.40999999999999998},"max":{"1min":1.135,"5min":1.2110000000000001,"15min":1.2470000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.88600000000000001,"15min":0.84899999999999998},"min":{"1min":0.66300000000000003,"5min":0.59699999999999998,"15min":0.505},"max":{"1min":0.96899999999999997,"5min":1.2829999999999999,"15min":1.4910000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91900000000000004,"15min":0.90700000000000003},"min":{"1min":0.84399999999999997,"5min":0.55000000000000004,"15min":0.45800000000000002},"max":{"1min":1.03,"5min":1.325,"15min":1.4530000000000001},"last":0.998}]},{"osd":2,"last update":"Wed Sep 18 12:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.78900000000000003,"15min":0.79100000000000004},"min":{"1min":0.626,"5min":0.55900000000000005,"15min":0.46400000000000002},"max":{"1min":0.92200000000000004,"5min":1.419,"15min":1.419},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91100000000000003,"15min":0.92000000000000004},"min":{"1min":0.65300000000000002,"5min":0.63600000000000001,"15min":0.441},"max":{"1min":1.0960000000000001,"5min":1.3280000000000001,"15min":1.357},"last":0.88800000000000001}]},{"osd":4,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68200000000000005,"15min":0.71099999999999997},"min":{"1min":0.50800000000000001,"5min":0.49299999999999999,"15min":0.43099999999999999},"max":{"1min":1.073,"5min":1.2569999999999999,"15min":1.5009999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.98899999999999999,"15min":0.95799999999999996},"min":{"1min":0.70099999999999996,"5min":0.60099999999999998,"15min":0.53700000000000003},"max":{"1min":1.202,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.77700000000000002}]},{"osd":5,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.65500000000000003,"5min":0.54500000000000004,"15min":0.34699999999999998},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.802},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0629999999999999,"15min":1.069},"min":{"1min":0.91300000000000003,"5min":0.68600000000000005,"15min":0.63200000000000001},"max":{"1min":1.3340000000000001,"5min":1.581,"15min":1.9039999999999999},"last":1.0209999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607796,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315060,"kb_used_data":9388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056780,"statfs":{"total":96636764160,"available":96314142720,"internally_reserved":0,"allocated":9613312,"data_stored":4954858,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.33100000000000002},"max":{"1min":0.71699999999999997,"5min":1.212,"15min":1.212},"last":0.497},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73099999999999998,"15min":0.72499999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.28699999999999998},"max":{"1min":0.85599999999999998,"5min":1.254,"15min":1.254},"last":0.60199999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.316},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.69099999999999995,"15min":0.67600000000000005},"min":{"1min":0.54100000000000004,"5min":0.498,"15min":0.41999999999999998},"max":{"1min":0.88500000000000001,"5min":1.577,"15min":4.1699999999999999},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.88500000000000001},"min":{"1min":0.63100000000000001,"5min":0.52500000000000002,"15min":0.36899999999999999},"max":{"1min":1.0229999999999999,"5min":1.3799999999999999,"15min":2.6389999999999998},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.86499999999999999,"15min":0.88500000000000001},"min":{"1min":0.66700000000000004,"5min":0.623,"15min":0.53400000000000003},"max":{"1min":1.069,"5min":1.2330000000000001,"15min":4.3310000000000004},"last":0.82699999999999996}]},{"osd":4,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.92500000000000004},"min":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.376},"max":{"1min":1.1579999999999999,"5min":1.2270000000000001,"15min":4.2270000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.997,"5min":0.98099999999999998,"15min":0.94899999999999995},"min":{"1min":0.89400000000000002,"5min":0.69599999999999995,"15min":0.55700000000000005},"max":{"1min":1.103,"5min":1.671,"15min":1.706},"last":1.01}]},{"osd":5,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.873,"15min":0.91500000000000004},"min":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.39000000000000001},"max":{"1min":1.0129999999999999,"5min":1.202,"15min":2.5960000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.93700000000000006,"15min":0.96499999999999997},"min":{"1min":0.82899999999999996,"5min":0.74299999999999999,"15min":0.65800000000000003},"max":{"1min":1.2330000000000001,"5min":1.4890000000000001,"15min":4.7590000000000003},"last":0.88200000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640500,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315112,"kb_used_data":9440,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056728,"statfs":{"total":96636764160,"available":96314089472,"internally_reserved":0,"allocated":9666560,"data_stored":5483786,"data_compressed":57055,"data_compressed_allocated":167936,"data_compressed_original":798720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55400000000000005,"15min":0.57199999999999995},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.313},"max":{"1min":0.76800000000000002,"5min":0.82899999999999996,"15min":1.2170000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.64400000000000002,"15min":0.67400000000000004},"min":{"1min":0.47199999999999998,"5min":0.442,"15min":0.39900000000000002},"max":{"1min":0.77700000000000002,"5min":0.95899999999999996,"15min":1.0369999999999999},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.63300000000000001,"15min":0.65200000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79600000000000004,"5min":0.94799999999999995,"15min":1.161},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.81200000000000006,"15min":0.77800000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.41799999999999998},"max":{"1min":0.90800000000000003,"5min":1.0660000000000001,"15min":1.1799999999999999},"last":0.94099999999999995}]},{"osd":3,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.83999999999999997,"15min":0.85199999999999998},"min":{"1min":0.69799999999999995,"5min":0.58099999999999996,"15min":0.53700000000000003},"max":{"1min":1.1200000000000001,"5min":1.2250000000000001,"15min":1.349},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.85899999999999999,"15min":0.89900000000000002},"min":{"1min":0.627,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.0069999999999999,"5min":1.157,"15min":1.486},"last":0.754}]},{"osd":4,"last update":"Wed Sep 18 12:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.83899999999999997,"15min":0.872},"min":{"1min":0.69799999999999995,"5min":0.58399999999999996,"15min":0.52000000000000002},"max":{"1min":1.0229999999999999,"5min":1.123,"15min":1.329},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.98799999999999999,"15min":0.94399999999999995},"min":{"1min":0.79700000000000004,"5min":0.71199999999999997,"15min":0.62},"max":{"1min":1.1379999999999999,"5min":1.262,"15min":1.4350000000000001},"last":1.0760000000000001}]},{"osd":5,"last update":"Wed Sep 18 12:04:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.79400000000000004,"5min":0.56100000000000005,"15min":0.504},"max":{"1min":1.054,"5min":1.3300000000000001,"15min":1.5609999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.997,"5min":1.0269999999999999,"15min":0.98399999999999999},"min":{"1min":0.82899999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.1719999999999999,"5min":1.53,"15min":1.752},"last":1.016}]}]},{"osd":0,"up_from":11,"seq":47244640500,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315068,"kb_used_data":9396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94056772,"statfs":{"total":96636764160,"available":96314134528,"internally_reserved":0,"allocated":9621504,"data_stored":4954882,"data_compressed":51698,"data_compressed_allocated":135168,"data_compressed_original":286720,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55400000000000005,"15min":0.57799999999999996},"min":{"1min":0.42099999999999999,"5min":0.34000000000000002,"15min":0.191},"max":{"1min":0.66600000000000004,"5min":2.4470000000000001,"15min":3.8849999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.73299999999999998,"15min":0.71299999999999997},"min":{"1min":0.49299999999999999,"5min":0.40300000000000002,"15min":0.26600000000000001},"max":{"1min":0.80500000000000005,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.78700000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.65200000000000002,"15min":0.67200000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.23300000000000001},"max":{"1min":0.85399999999999998,"5min":1.3799999999999999,"15min":3.6030000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.86599999999999999,"15min":0.82299999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.30399999999999999},"max":{"1min":1.2749999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85199999999999998,"15min":0.84899999999999998},"min":{"1min":0.65700000000000003,"5min":0.60099999999999998,"15min":0.28100000000000003},"max":{"1min":1.0589999999999999,"5min":2.2440000000000002,"15min":3.7909999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.93100000000000005,"15min":0.92200000000000004},"min":{"1min":0.71999999999999997,"5min":0.65800000000000003,"15min":0.313},"max":{"1min":1.0669999999999999,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":0.88600000000000001}]},{"osd":4,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.028,"15min":0.96299999999999997},"min":{"1min":0.753,"5min":0.68600000000000005,"15min":0.40799999999999997},"max":{"1min":1.1950000000000001,"5min":2.8530000000000002,"15min":3.6709999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.022,"5min":0.99399999999999999,"15min":1},"min":{"1min":0.86499999999999999,"5min":0.67400000000000004,"15min":0.33700000000000002},"max":{"1min":1.165,"5min":2.726,"15min":3.5270000000000001},"last":0.99299999999999999}]},{"osd":5,"last update":"Wed Sep 18 12:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.89400000000000002,"15min":0.88800000000000001},"min":{"1min":0.75900000000000001,"5min":0.66700000000000004,"15min":0.36799999999999999},"max":{"1min":1.0880000000000001,"5min":1.389,"15min":3.6469999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.98799999999999999,"15min":0.97199999999999998},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.39500000000000002},"max":{"1min":1.347,"5min":2.7429999999999999,"15min":4.3280000000000003},"last":0.94499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5357,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:24.488 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-18T12:04:24.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient: get_monmap_and_config 2024-09-18T12:04:24.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient: build_initial_monmap 2024-09-18T12:04:24.593 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient: monmap: 2024-09-18T12:04:24.593 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:04:24.593 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.596 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T12:04:24.594321+0000 2024-09-18T12:04:24.596 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T12:04:24.594321+0000 2024-09-18T12:04:24.596 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:04:24.596 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.596 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:04:24.596 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-09-18T12:04:24.596 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-b 2024-09-18T12:04:24.597 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 1 Processor -- start 2024-09-18T12:04:24.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 1 -- start start 2024-09-18T12:04:24.600 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient: init 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient: _reopen_session rank -1 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient(hunting): picked mon.noname-a con 0x7fde1409d960 addr v1:172.21.15.112:6789/0 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient(hunting): picked mon.noname-c con 0x7fde14131740 addr v1:172.21.15.129:6789/0 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient(hunting): picked mon.noname-b con 0x7fde14135240 addr v1:172.21.15.112:6790/0 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1409dd30 con 0x7fde1409d960 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde14064e10 con 0x7fde14131740 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1406f9f0 con 0x7fde14135240 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient(hunting): _renew_subs 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1acbb700 10 monclient(hunting): authenticate will time out at 2036.435091s 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1a4ba700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fde14135240 0x7fde14137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50148/0 (socket says 172.21.15.112:50148) 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.592+0000 7fde1a4ba700 1 -- 172.21.15.112:0/632954516 learned_addr learned my addr 172.21.15.112:0/632954516 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde19cb9700 1 -- 172.21.15.112:0/632954516 >> v1:172.21.15.112:6789/0 conn(0x7fde1409d960 legacy=0x7fde140955a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.601 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1a4ba700 1 -- 172.21.15.112:0/632954516 >> v1:172.21.15.112:6790/0 conn(0x7fde14135240 legacy=0x7fde14137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde194b8700 1 -- 172.21.15.112:0/632954516 >> v1:172.21.15.129:6789/0 conn(0x7fde14131740 legacy=0x7fde14133b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634654563 0 0) 0x7fde1409dd30 con 0x7fde1409d960 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): my global_id is 164250 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde00003570 con 0x7fde1409d960 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497195244 0 0) 0x7fde1406f9f0 con 0x7fde14135240 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): my global_id is 154346 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1409dd30 con 0x7fde14135240 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092692524 0 0) 0x7fde14064e10 con 0x7fde14131740 2024-09-18T12:04:24.602 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient(hunting): my global_id is 144433 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1406f9f0 con 0x7fde14131740 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 389941153 0 0) 0x7fde00003570 con 0x7fde1409d960 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 >> v1:172.21.15.112:6790/0 conn(0x7fde14135240 legacy=0x7fde14137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 >> v1:172.21.15.129:6789/0 conn(0x7fde14131740 legacy=0x7fde14133b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: _finish_hunting 0 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 monclient: found mon.noname-a 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.112:6789/0 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde14099b30 con 0x7fde1409d960 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: _finish_auth 0 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:04:24.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fde04001990 con 0x7fde1409d960 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 5 monclient: authenticate success, global_id 164250 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: dump: 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.604 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 <== mon.0 v1:172.21.15.112:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde04001c20 con 0x7fde1409d960 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 10 monclient: get_monmap_and_config success 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 4 set_mon_vals no callback set 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 1 -- 172.21.15.112:0/632954516 <== mon.0 v1:172.21.15.112:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fde040020a0 con 0x7fde1409d960 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde18cb7700 10 monclient: dump: 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.605 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 10 monclient: shutdown 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 1 -- 172.21.15.112:0/632954516 >> v1:172.21.15.112:6789/0 conn(0x7fde1409d960 legacy=0x7fde140955a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 1 -- 172.21.15.112:0/632954516 shutdown_connections 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 1 -- 172.21.15.112:0/632954516 >> 172.21.15.112:0/632954516 conn(0x7fde140a15d0 msgr2=0x7fde140a19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 1 -- 172.21.15.112:0/632954516 shutdown_connections 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.596+0000 7fde1acbb700 1 -- 172.21.15.112:0/632954516 wait complete. 2024-09-18T12:04:24.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient: build_initial_monmap 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient: monmap: 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 0 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 0.000000 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:created 0.000000 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 0 (unknown) 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.noname-a 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.noname-c 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 librados: starting msgr at 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 librados: starting objecter 2024-09-18T12:04:24.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 Processor -- start 2024-09-18T12:04:24.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 -- start start 2024-09-18T12:04:24.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 librados: setting wanted keys 2024-09-18T12:04:24.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 librados: calling monclient init 2024-09-18T12:04:24.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient: init 2024-09-18T12:04:24.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient: _reopen_session rank -1 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient(hunting): picked mon.noname-a con 0x7fde1409d960 addr v1:172.21.15.112:6789/0 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient(hunting): picked mon.noname-c con 0x7fde14131740 addr v1:172.21.15.112:6790/0 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient(hunting): picked mon.noname-b con 0x7fde14135240 addr v1:172.21.15.129:6789/0 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1406e930 con 0x7fde1409d960 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde14134c70 con 0x7fde14135240 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 1 -- --> v1:172.21.15.112:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1406fc00 con 0x7fde14131740 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient(hunting): _renew_subs 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde1acbb700 10 monclient(hunting): authenticate will time out at 2036.442397s 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde194b8700 1 --1- >> v1:172.21.15.112:6790/0 conn(0x7fde14131740 0x7fde1409cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6790/0 says I am v1:172.21.15.112:50164/0 (socket says 172.21.15.112:50164) 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde19cb9700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fde1409d960 0x7fde1409c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59700/0 (socket says 172.21.15.112:59700) 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.600+0000 7fde194b8700 1 -- 172.21.15.112:0/2521386600 learned_addr learned my addr 172.21.15.112:0/2521386600 (peer_addr_for_me v1:172.21.15.112:0/0) 2024-09-18T12:04:24.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde194b8700 1 -- 172.21.15.112:0/2521386600 >> v1:172.21.15.112:6790/0 conn(0x7fde14131740 legacy=0x7fde1409cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 client.?.objecter ms_handle_connect 0x7fde14131740 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 client.?.objecter ms_handle_connect 0x7fde1409d960 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde19cb9700 1 -- 172.21.15.112:0/2521386600 >> v1:172.21.15.112:6789/0 conn(0x7fde1409d960 legacy=0x7fde1409c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 client.?.objecter ms_handle_connect 0x7fde14135240 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 client.?.objecter resend_mon_ops 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1a4ba700 1 -- 172.21.15.112:0/2521386600 >> v1:172.21.15.129:6789/0 conn(0x7fde14135240 legacy=0x7fde14147f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953919425 0 0) 0x7fde1406fc00 con 0x7fde14131740 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): my global_id is 154349 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf8004f10 con 0x7fde14131740 2024-09-18T12:04:24.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285377725 0 0) 0x7fde1406e930 con 0x7fde1409d960 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): my global_id is 164253 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1406fc00 con 0x7fde1409d960 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742790279 0 0) 0x7fde14134c70 con 0x7fde14135240 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): my global_id is 144436 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1406e930 con 0x7fde14135240 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3316191766 0 0) 0x7fddf8004f10 con 0x7fde14131740 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde14134c70 con 0x7fde14131740 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fde10001c40 con 0x7fde14131740 2024-09-18T12:04:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): dump: 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.616 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1074534159 0 0) 0x7fde1406fc00 con 0x7fde1409d960 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddf8004f10 con 0x7fde1409d960 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fde040020a0 con 0x7fde1409d960 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): dump: 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.617 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3004212139 0 0) 0x7fde1406e930 con 0x7fde14135240 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde1406fc00 con 0x7fde14135240 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fde0c001ef0 con 0x7fde14135240 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-18T12:04:24.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient(hunting): dump: 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4165749190 0 0) 0x7fde14134c70 con 0x7fde14131740 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 >> v1:172.21.15.129:6789/0 conn(0x7fde14135240 legacy=0x7fde14147f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 >> v1:172.21.15.112:6789/0 conn(0x7fde1409d960 legacy=0x7fde1409c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient: _finish_hunting 0 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 monclient: found mon.c 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1406e700 con 0x7fde14131740 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient: _finish_auth 0 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 5 monclient: authenticate success, global_id 154349 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 10 monclient: _renew_subs 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1406c750 con 0x7fde14131740 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 10 client.154349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 10 monclient: _renew_subs 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:04:24.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1406e140 con 0x7fde14131740 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde1acbb700 1 librados: init done 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde100022b0 con 0x7fde14131740 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1188057296 0 0) 0x7fde10002600 con 0x7fde14131740 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde0bfff700 4 set_mon_vals no callback set 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 10 monclient: dump: 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:epoch 1 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:fsid e68e10b8-0874-4c25-87a0-f38d9ca2e0d2 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:last_changed 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:created 2024-09-18T11:43:39.500070+0000 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:min_mon_release 17 (quincy) 2024-09-18T12:04:24.621 INFO:teuthology.orchestra.run.smithi112.stderr:election_strategy: 1 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:0: v1:172.21.15.112:6789/0 mon.a 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2: v1:172.21.15.112:6790/0 mon.c 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.604+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 7 ==== mgrmap(e 5) v1 ==== 42763+0+0 (unknown 2549215517 0 0) 0x7fde10002760 con 0x7fde14131740 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 8 ==== osd_map(699..699 src has 1..699) v4 ==== 4633+0+0 (unknown 2740130809 0 0) 0x7fde10043c00 con 0x7fde14131740 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde09ffb700 10 client.154349.objecter ms_dispatch 0x7fde14000fd0 osd_map(699..699 src has 1..699) v4 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde09ffb700 3 client.154349.objecter handle_osd_map got epochs [699,699] > 0 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde09ffb700 3 client.154349.objecter handle_osd_map decoding full epoch 699 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde09ffb700 20 client.154349.objecter dump_active .. 0 homeless 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde19cb9700 10 monclient: get_auth_request con 0x7fddf8038550 auth_method 0 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde1acbb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:24.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde1acbb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:24.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde1acbb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:04:24.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf40050b0 con 0x7fde14131740 2024-09-18T12:04:24.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.608+0000 7fde09ffb700 10 client.154349.objecter ms_handle_connect 0x7fddf8038550 2024-09-18T12:04:24.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.616+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 1092875540 0 638601925) 0x7fde100146c0 con 0x7fde14131740 2024-09-18T12:04:24.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.616+0000 7fde09ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:04:24.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.616+0000 7fde09ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T12:04:24.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.776+0000 7fde1acbb700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T12:04:24.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.776+0000 7fde1acbb700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T12:04:24.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.776+0000 7fde1acbb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.112:6790/0 2024-09-18T12:04:24.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.776+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 --> v1:172.21.15.112:6790/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fddf40058e0 con 0x7fde14131740 2024-09-18T12:04:24.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.788+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 10 ==== config(1 keys) v1 ==== 35+0+0 (unknown 2555425225 0 0) 0x7fde100146c0 con 0x7fde14131740 2024-09-18T12:04:24.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.788+0000 7fde09ffb700 10 monclient: handle_config config(1 keys) v1 2024-09-18T12:04:24.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.788+0000 7fde0bfff700 4 set_mon_vals no callback set 2024-09-18T12:04:24.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.788+0000 7fde0bfff700 10 set_mon_vals mon_health_to_clog = false 2024-09-18T12:04:24.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.788+0000 7fde09ffb700 1 -- 172.21.15.112:0/2521386600 <== mon.2 v1:172.21.15.112:6790/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (unknown 691139944 0 0) 0x7fde100144e0 con 0x7fde14131740 2024-09-18T12:04:24.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.788+0000 7fde09ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T12:04:24.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.788+0000 7fde09ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T12:04:24.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 10 librados: watch_flush enter 2024-09-18T12:04:24.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 10 librados: watch_flush exit 2024-09-18T12:04:24.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 20 client.154349.objecter shutdown clearing up homeless session... 2024-09-18T12:04:24.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 10 client.154349.objecter successfully canceled tick 2024-09-18T12:04:24.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 >> v1:172.21.15.129:6812/20806 conn(0x7fddf8038550 legacy=0x7fddf803aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 10 monclient: shutdown 2024-09-18T12:04:24.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:04:24.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 >> v1:172.21.15.112:6790/0 conn(0x7fde14131740 legacy=0x7fde1409cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:04:24.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 shutdown_connections 2024-09-18T12:04:24.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 >> 172.21.15.112:0/2521386600 conn(0x7fde140a15d0 msgr2=0x7fde14130c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:04:24.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 shutdown_connections 2024-09-18T12:04:24.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 1 -- 172.21.15.112:0/2521386600 wait complete. 2024-09-18T12:04:24.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-18T12:04:24.792+0000 7fde1acbb700 1 librados: shutdown 2024-09-18T12:04:24.805 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T12:04:24.805 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T12:04:24.805 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T12:04:24.805 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:24.873 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T12:04:24.874 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T12:04:24.874 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:24.960 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T12:04:24.960 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T12:04:24.960 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:25.036 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T12:04:25.036 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T12:04:25.036 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:25.106 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T12:04:25.106 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-18T12:04:25.106 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:25.195 INFO:tasks.ceph.osd.4:Stopped 2024-09-18T12:04:25.196 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-18T12:04:25.196 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:25.256 INFO:tasks.ceph.osd.5:Stopped 2024-09-18T12:04:25.256 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T12:04:25.256 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-18T12:04:25.256 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:25.288 INFO:tasks.ceph.mgr.x:Stopped 2024-09-18T12:04:25.288 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T12:04:25.288 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:04:25.288 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:25.307 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:04:25.307 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:04:25.307 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:25.323 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:04:25.323 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T12:04:25.323 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:04:25.345 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T12:04:25.345 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T12:04:25.346 DEBUG:teuthology.orchestra.run.smithi112:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-18T12:04:25.370 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T12:04:25.370 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-18T12:04:25.699 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T12:04:25.699 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-18T12:04:25.922 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T12:04:25.922 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-18T12:04:26.095 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T12:04:26.095 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-18T12:04:26.306 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T12:04:26.306 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-18T12:04:26.562 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T12:04:26.562 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-18T12:04:26.746 INFO:tasks.ceph:Archiving mon data... 2024-09-18T12:04:26.748 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/data/mon.a.tgz 2024-09-18T12:04:26.748 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-18T12:04:26.755 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.Dv6WuDg7ZI 2024-09-18T12:04:26.755 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Dv6WuDg7ZI 2024-09-18T12:04:26.904 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.Dv6WuDg7ZI 2024-09-18T12:04:26.924 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.Dv6WuDg7ZI is 724KB 2024-09-18T12:04:26.993 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.Dv6WuDg7ZI 2024-09-18T12:04:27.000 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/data/mon.c.tgz 2024-09-18T12:04:27.000 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-18T12:04:27.006 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.yNHVczQVtL 2024-09-18T12:04:27.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.yNHVczQVtL 2024-09-18T12:04:27.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.yNHVczQVtL 2024-09-18T12:04:27.222 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.yNHVczQVtL is 723KB 2024-09-18T12:04:27.249 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.yNHVczQVtL 2024-09-18T12:04:27.255 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/data/mon.b.tgz 2024-09-18T12:04:27.255 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-18T12:04:27.262 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.J9Cy3WSy7B 2024-09-18T12:04:27.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.J9Cy3WSy7B 2024-09-18T12:04:27.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.J9Cy3WSy7B 2024-09-18T12:04:27.481 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.J9Cy3WSy7B is 724KB 2024-09-18T12:04:27.549 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.J9Cy3WSy7B 2024-09-18T12:04:27.556 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T12:04:27.557 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T12:04:27.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T12:04:27.573 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-18T12:04:27.573 DEBUG:teuthology.orchestra.run.smithi112:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T12:04:27.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T12:04:27.594 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T12:04:27.642 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T12:04:27.648 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T12:04:27.650 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/remote/smithi112/crash 2024-09-18T12:04:27.651 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T12:04:27.668 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/remote/smithi129/crash 2024-09-18T12:04:27.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T12:04:27.685 INFO:tasks.ceph:Compressing logs... 2024-09-18T12:04:27.686 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T12:04:27.688 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T12:04:27.708 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25846.log 2024-09-18T12:04:27.709 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28950.log 2024-09-18T12:04:27.709 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23878.log 2024-09-18T12:04:27.709 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25846.log.gz 2024-09-18T12:04:27.710 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26285.log 2024-09-18T12:04:27.710 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23642.log 2024-09-18T12:04:27.712 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23878.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28950.log.gz 2024-09-18T12:04:27.712 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25058.log 2024-09-18T12:04:27.712 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.23878.log.gz 2024-09-18T12:04:27.712 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27875.log 2024-09-18T12:04:27.712 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.26285.log.gz 2024-09-18T12:04:27.712 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26516.log 2024-09-18T12:04:27.713 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25058.log: /var/log/ceph/ceph-client.admin.27875.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27875.log.gz 2024-09-18T12:04:27.713 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.23642.log.gz 2024-09-18T12:04:27.713 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.25058.log.gz 2024-09-18T12:04:27.713 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-18T12:04:27.714 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24236.log 2024-09-18T12:04:27.714 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.26516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24263.log 2024-09-18T12:04:27.715 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24236.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.26516.log.gz 2024-09-18T12:04:27.715 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25114.log 2024-09-18T12:04:27.716 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.24236.log.gz 2024-09-18T12:04:27.716 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24757.log 2024-09-18T12:04:27.717 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26998.log 2024-09-18T12:04:27.718 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25114.log: /var/log/ceph/ceph-client.admin.24757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27079.log 2024-09-18T12:04:27.718 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.24263.log.gz 2024-09-18T12:04:27.718 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25114.log.gz 2024-09-18T12:04:27.718 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24757.log.gz 2024-09-18T12:04:27.719 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26998.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26998.log.gz 2024-09-18T12:04:27.719 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24425.log 2024-09-18T12:04:27.719 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27079.log: /var/log/ceph/ceph-client.admin.25603.log 2024-09-18T12:04:27.720 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26039.log 2024-09-18T12:04:27.721 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24425.log: /var/log/ceph/ceph-client.admin.25603.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27079.log.gz 2024-09-18T12:04:27.721 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25684.log 2024-09-18T12:04:27.721 INFO:teuthology.orchestra.run.smithi112.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.25603.log.gz 2024-09-18T12:04:27.721 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24425.log.gz 2024-09-18T12:04:27.722 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26039.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26039.log.gz 2024-09-18T12:04:27.722 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28420.log 2024-09-18T12:04:27.723 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28283.log 2024-09-18T12:04:27.723 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24923.log 2024-09-18T12:04:27.724 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25684.log.gz 2024-09-18T12:04:27.727 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24344.log 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23669.log 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24923.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28420.log.gz 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28283.log.gz 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27956.log 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24923.log.gz 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29274.log 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23669.log: /var/log/ceph/ceph-client.admin.27956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23877.log 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.24344.log.gz 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25819.log 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27956.log.gz 2024-09-18T12:04:27.728 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.23669.log.gz 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27326.log 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28474.log 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29274.log.gz 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25819.log: /var/log/ceph/ceph-client.admin.27326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24867.log 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.23877.log.gz 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.27326.log.gz 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26258.log 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28474.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25819.log.gz 2024-09-18T12:04:27.729 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29058.log 2024-09-18T12:04:27.750 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26917.log 2024-09-18T12:04:27.750 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28474.log.gz 2024-09-18T12:04:27.750 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29058.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.24867.log.gz 2024-09-18T12:04:27.750 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26447.log 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26258.log.gz 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23873.log 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.29058.log.gz 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28896.log 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.26917.log.gz 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.26447.log.gz 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.27556.log 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23873.log: /var/log/ceph/ceph-client.admin.28896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27419.log 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27610.log 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27556.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.23873.log.gz 2024-09-18T12:04:27.751 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27419.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28896.log.gz 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29112.log 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27419.log.gz 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27214.log 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27610.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27556.log.gz 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23714.log 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27902.log 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27214.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27610.log.gz 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24838.log 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27214.log.gz 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25435.log 2024-09-18T12:04:27.752 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29112.log.gz 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.23714.log.gz 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27902.log: /var/log/ceph/ceph-client.admin.24838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26231.log 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24896.log 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25435.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27902.log.gz 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.24838.log.gz 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25381.log 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.25435.log.gz 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25983.log 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26231.log.gz 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28869.log 2024-09-18T12:04:27.753 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25381.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24896.log.gz 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28842.log 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25630.log 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3%/var/log/ceph/ceph-client.admin.28869.log: -- replaced with /var/log/ceph/ceph-client.admin.25381.log.gz 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24209.log 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28842.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25983.log.gz 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26312.log 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28869.log.gz 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28842.log.gz 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25287.log 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24209.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25630.log.gz 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26204.log 2024-09-18T12:04:27.754 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26653.log 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25287.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.24209.log.gz 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25576.log 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26204.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.26312.log.gz 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24950.log 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.25287.log.gz 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26653.log: /var/log/ceph/ceph-client.admin.25576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24078.log 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26204.log.gz 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.26653.log.gz-5 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr:--verbose -- /var/log/ceph/ceph-client.admin.24371.log 2024-09-18T12:04:27.755 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24950.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.25576.log.gz 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27446.log 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27983.log 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24371.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23876.log 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24950.log.gz 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27446.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.24078.log.gz 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27983.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.24371.log.gz 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24082.log 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27446.log.gz 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29301.log 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23876.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27983.log.gz 2024-09-18T12:04:27.756 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24506.log 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24082.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.23876.log.gz 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23875.log 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25354.log 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24506.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.24082.log.gz 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27529.log 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29301.log.gz 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26010.log 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25354.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.23875.log.gz 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24506.log.gz 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26734.log 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27529.log: /var/log/ceph/ceph-client.admin.26010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25873.log 2024-09-18T12:04:27.757 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.25354.log.gz 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26366.log 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26734.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27529.log.gz 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28923.log 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25873.log: -- replaced with /var/log/ceph/ceph-client.admin.26010.log.gz 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26734.log.gz 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27475.log 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26366.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25873.log.gz 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26066.log 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27160.log 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27475.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26366.log.gz 2024-09-18T12:04:27.758 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28923.log.gz 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24077.log 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24317.log 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7%/var/log/ceph/ceph-client.admin.27160.log: -- replaced with /var/log/ceph/ceph-client.admin.27475.log.gz 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29220.log 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0%/var/log/ceph/ceph-client.admin.24077.log: -- replaced with /var/log/ceph/ceph-client.admin.26066.log.gz 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28037.log 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27160.log.gz 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.24077.log.gz 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29220.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.24317.log.gz 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27241.log 2024-09-18T12:04:27.759 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.29220.log.gz/var/log/ceph/ceph-mon.c.log: --verbose 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:-- /var/log/ceph/ceph.audit.log 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28037.log.gz 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23565.log 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26393.log 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.audit.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27241.log.gz 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28788.log 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23565.log: gzip /var/log/ceph/ceph-client.admin.26393.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25725.log 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28393.log 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28788.log: /var/log/ceph/ceph-client.admin.25725.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26393.log.gz 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.23565.log.gz 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28788.log.gz 2024-09-18T12:04:27.760 INFO:teuthology.orchestra.run.smithi112.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25725.log.gz 2024-09-18T12:04:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25956.log 2024-09-18T12:04:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23615.log 2024-09-18T12:04:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23712.log 2024-09-18T12:04:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25956.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28393.log.gz 2024-09-18T12:04:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28189.log 2024-09-18T12:04:27.761 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23712.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.23615.log.gz 2024-09-18T12:04:27.761 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.25956.log.gz 2024-09-18T12:04:27.761 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.23712.log.gz 2024-09-18T12:04:27.779 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24318.log 2024-09-18T12:04:27.779 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23975.log 2024-09-18T12:04:27.779 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24825.log 2024-09-18T12:04:27.779 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25018.log 2024-09-18T12:04:27.779 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23975.log: /var/log/ceph/ceph-client.admin.24825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24424.log 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.25018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25101.log 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.24318.log.gz 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24424.log: /var/log/ceph/ceph-client.admin.25101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23778.log 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23818.log 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.23778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25102.log 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.25018.log.gz 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23818.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.24825.log.gz 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.25101.log.gz 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24723.log 2024-09-18T12:04:27.780 INFO:teuthology.orchestra.run.smithi129.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24424.log.gz 2024-09-18T12:04:27.781 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24775.log 2024-09-18T12:04:27.781 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.25102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24283.log 2024-09-18T12:04:27.781 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24723.log: /var/log/ceph/ceph-client.admin.24775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24828.log 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24283.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.24723.log.gz 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24722.log 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24828.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.23818.log.gz 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr: 90.8% 87.0% -- replaced with /var/log/ceph/ceph-client.admin.24828.log.gz 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25102.log.gz 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.24775.log.gz 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24209.log 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23869.log 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24722.log: /var/log/ceph/ceph-client.admin.24209.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.24283.log.gz 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24125.log 2024-09-18T12:04:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.23778.log.gz/var/log/ceph/ceph-client.admin.23869.log: 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24230.log 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.24722.log.gz 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20931.log 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24230.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.24125.log.gz 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.24209.log.gz 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24426.log 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.20931.log: /var/log/ceph/ceph-client.admin.24426.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24230.log.gz 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.20931.log.gz 2024-09-18T12:04:27.793 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24426.log.gz 2024-09-18T12:04:27.798 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24826.log 2024-09-18T12:04:27.798 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20986.log 2024-09-18T12:04:27.798 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T12:04:27.798 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24826.log: /var/log/ceph/ceph-client.admin.20986.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.24826.log.gz 2024-09-18T12:04:27.798 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.20986.log.gz 2024-09-18T12:04:27.798 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24060.log 2024-09-18T12:04:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28092.log 2024-09-18T12:04:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23537.log 2024-09-18T12:04:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26545.log 2024-09-18T12:04:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28092.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28189.log.gz 2024-09-18T12:04:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25087.log 2024-09-18T12:04:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23537.log: /var/log/ceph/ceph-client.admin.26545.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28092.log.gz 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27392.log 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27819.log 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26545.log.gz 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25087.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23537.log.gz 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27392.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25087.log.gz 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27392.log.gz 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26599.log 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17252.log 2024-09-18T12:04:27.803 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27846.log 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28010.log 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.tmp-client.admin.17252.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17252.log.gz 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27819.log.gz 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28256.log 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27846.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26599.log.gz 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28815.log 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28734.log 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26707.log 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27846.log.gz 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28815.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28256.log.gz 2024-09-18T12:04:27.804 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27133.log 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28734.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28010.log.gz 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29031.log 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26707.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28815.log.gz 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26944.log 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27133.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28734.log.gz 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26707.log.gz 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27929.log 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24811.log 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29031.log.gz 2024-09-18T12:04:27.805 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27133.log.gz 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27025.log 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27929.log: /var/log/ceph/ceph-client.admin.24811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23716.log 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26944.log.gz 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26761.log 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24811.log.gz 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27929.log.gz 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24024.log 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27025.log.gz 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23716.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-18T12:04:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26761.log: /var/log/ceph/ceph-client.admin.24024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29139.log 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.23716.log.gz 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27792.log 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26761.log.gz 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29139.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.24024.log.gz 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26420.log 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24020.log 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27792.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.29139.log.gz 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27052.log 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24452.log 2024-09-18T12:04:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24020.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27792.log.gz 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.26420.log.gz 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28447.log 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26680.log 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27052.log.gz -- replaced with /var/log/ceph/ceph-client.admin.24020.log.gz 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24452.log: /var/log/ceph/ceph-client.admin.28447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23511.log 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24452.log.gz 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26680.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28447.log.gz 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26680.log.gz 2024-09-18T12:04:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29193.log 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23511.log: /var/log/ceph/ceph-client.admin.26971.log 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24398.log 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24182.log 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.23511.log.gz 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26971.log: /var/log/ceph/ceph-client.admin.24398.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29193.log.gz 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24398.log.gz 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26971.log.gz 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23710.log 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28582.log 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24182.log: /var/log/ceph/ceph-client.admin.23710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28501.log 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.24182.log.gz 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25031.log 2024-09-18T12:04:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25657.log 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.23710.log.gz 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28501.log: /var/log/ceph/ceph-client.admin.25031.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28582.log.gz 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28501.log.gz 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.25031.log.gz 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26177.log 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27268.log 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25408.log 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25495.log 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25657.log.gz 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27268.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.26177.log.gz 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27187.log 2024-09-18T12:04:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26474.log 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27268.log.gz 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25495.log: /var/log/ceph/ceph-client.admin.27187.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25408.log.gz 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.25495.log.gz 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.27187.log.gz 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26339.log 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24079.log 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28707.log 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26339.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26474.log.gz 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24977.log 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24079.log: /var/log/ceph/ceph-client.admin.28707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27583.log 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26339.log.gz 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.24079.log.gz 2024-09-18T12:04:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24977.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28707.log.gz 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24977.log.gz 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26788.log 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27583.log: /var/log/ceph/ceph-client.admin.27106.log 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29166.log 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27295.log 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27106.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27583.log.gz 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.26788.log.gz 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29166.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27106.log.gz 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29166.log.gz 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24479.log 2024-09-18T12:04:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25900.log 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29085.log /var/log/ceph/ceph-client.admin.24479.log: 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24587.log 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27295.log.gz/var/log/ceph/ceph-client.admin.25900.log: 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28065.log 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.24479.log.gz 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26572.log 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25900.log.gz 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24587.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29085.log.gz 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28528.log 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28761.log 2024-09-18T12:04:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26572.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24587.log.gz 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28065.log.gz 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28528.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26572.log.gz 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28528.log.gz 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28146.log 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24560.log 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29353.log 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28146.log: /var/log/ceph/ceph-client.admin.24560.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28761.log.gz 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25792.log 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28146.log.gz 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.24560.log.gz 2024-09-18T12:04:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24688.log 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20983.log 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25792.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29353.log.gz 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29004.log 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24688.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25792.log.gz 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25522.log 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.20983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25004.log 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr: 86.7%/var/log/ceph/ceph-client.admin.29004.log: -- replaced with /var/log/ceph/ceph-client.admin.24688.log.gz 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21071.log 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25522.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T12:04:27.815 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/ceph-client.admin.20983.log.gz 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29004.log.gz 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25004.log: /var/log/ceph/ceph-client.admin.21071.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25522.log.gz 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.25004.log.gz 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25466.log 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.21071.log.gz 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27718.log 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28337.log 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25549.log 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9%/var/log/ceph/ceph-client.admin.28337.log: -- replaced with /var/log/ceph/ceph-client.admin.25466.log.gz 2024-09-18T12:04:27.816 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27718.log.gz 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28337.log.gz-5 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:--verbose -- /var/log/ceph/ceph-client.admin.27691.log 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23713.log 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27691.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.25549.log.gz 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.0.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27691.log.gz 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20863.log 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23874.log 2024-09-18T12:04:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23713.log: /var/log/ceph/ceph-client.admin.20863.log: 88.1% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.20863.log.gz -- replaced with /var/log/ceph/ceph-client.admin.23713.log.gz 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26842.log 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29247.log 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23874.log: /var/log/ceph/ceph-client.admin.26842.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.23874.log.gz 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.26842.log.gz 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28977.log 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28119.log 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29247.log: /var/log/ceph/ceph-client.admin.28977.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.29247.log.gz 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26815.log 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28977.log.gz 2024-09-18T12:04:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28119.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28119.log.gz 2024-09-18T12:04:27.828 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25015.log 2024-09-18T12:04:27.828 INFO:teuthology.orchestra.run.smithi129.stderr: 91.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T12:04:27.828 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24060.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.24060.log.gz 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23789.log 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.25015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23806.log 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24023.log 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23806.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.25015.log.gz 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24724.log 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24023.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.23789.log.gz 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25100.log 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24724.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.24023.log.gz 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23859.log 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.25100.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24724.log.gz 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25016.log 2024-09-18T12:04:27.829 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23859.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.25100.log.gz 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25019.log 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.25016.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.23859.log.gz 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24000.log 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.25019.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.25016.log.gz 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24037.log 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24000.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.25019.log.gz 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24827.log 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24037.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.24000.log.gz 2024-09-18T12:04:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24829.log 2024-09-18T12:04:27.833 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28366.log 2024-09-18T12:04:27.833 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29328.log 2024-09-18T12:04:27.833 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28366.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.26815.log.gz 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr: 89.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28366.log.gz 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27502.log 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24784.log 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.29328.log: gzip -5 --verbose /var/log/ceph/ceph-client.admin.27502.log: -- /var/log/ceph/ceph-client.admin.27664.log 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28555.log 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24784.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29328.log.gz 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27502.log.gz 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27745.log 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24784.log.gz 2024-09-18T12:04:27.834 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25927.log 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28555.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.27664.log.gz 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24290.log 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28555.log.gz 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.25927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26626.log 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27745.log.gz 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24533.log 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24290.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25927.log.gz 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.26626.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.24290.log.gz 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28680.log 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.26626.log.gz 2024-09-18T12:04:27.835 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23715.log 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28680.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.24533.log.gz 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28680.log.gz 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28613.log 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24641.log 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.23715.log: /var/log/ceph/ceph-client.admin.28613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27637.log 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.24641.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.23715.log.gz 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28613.log.gz 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28310.log 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-18T12:04:27.836 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.27637.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.24641.log.gz 2024-09-18T12:04:27.837 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-client.admin.28310.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27637.log.gz 2024-09-18T12:04:27.837 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28310.log.gz 2024-09-18T12:04:27.837 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24614.log 2024-09-18T12:04:27.837 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.24614.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.24614.log.gz 2024-09-18T12:04:27.838 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24827.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.24827.log.gz 2024-09-18T12:04:27.838 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23765.log 2024-09-18T12:04:27.838 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24829.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24829.log.gz 2024-09-18T12:04:27.839 INFO:teuthology.orchestra.run.smithi129.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.24037.log.gz 2024-09-18T12:04:27.839 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24415.log 2024-09-18T12:04:27.839 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24773.log 2024-09-18T12:04:27.839 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24415.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.24415.log.gz 2024-09-18T12:04:27.842 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23945.log 2024-09-18T12:04:27.845 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24773.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24773.log.gz 2024-09-18T12:04:27.846 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23773.log 2024-09-18T12:04:27.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23945.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.23765.log.gz 2024-09-18T12:04:27.861 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24428.log 2024-09-18T12:04:27.866 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23773.log: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.23806.log.gz 2024-09-18T12:04:27.874 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20903.log 2024-09-18T12:04:27.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24097.log 2024-09-18T12:04:27.876 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.20903.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.24428.log.gz 2024-09-18T12:04:27.876 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.20903.log.gz 2024-09-18T12:04:27.882 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T12:04:27.885 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24097.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.24097.log.gz 2024-09-18T12:04:27.886 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24373.log 2024-09-18T12:04:27.896 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.log: 85.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24774.log 2024-09-18T12:04:27.897 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24373.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.24373.log.gz 2024-09-18T12:04:27.897 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T12:04:27.898 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24427.log 2024-09-18T12:04:27.900 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24774.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24774.log.gz 2024-09-18T12:04:27.900 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24150.log 2024-09-18T12:04:27.903 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24427.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24427.log.gz 2024-09-18T12:04:27.903 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21013.log 2024-09-18T12:04:27.910 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24150.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.24150.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-18T12:04:27.910 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T12:04:27.911 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.21013.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.21013.log.gz 2024-09-18T12:04:27.912 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24195.log 2024-09-18T12:04:27.925 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.4.log: 94.9% -- replaced with /var/log/ceph/ceph-client.admin.23945.log.gz 2024-09-18T12:04:27.925 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20878.log 2024-09-18T12:04:27.932 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24195.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.24195.log.gz 2024-09-18T12:04:27.932 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24306.log 2024-09-18T12:04:27.933 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.20878.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.20878.log.gz 2024-09-18T12:04:27.933 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20958.log 2024-09-18T12:04:27.939 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24306.log: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.24306.log.gz 2024-09-18T12:04:27.939 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24206.log 2024-09-18T12:04:27.940 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.20958.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.20958.log.gz 2024-09-18T12:04:27.940 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-18T12:04:27.948 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24206.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24206.log.gz 2024-09-18T12:04:27.948 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24414.log 2024-09-18T12:04:27.962 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23794.log 2024-09-18T12:04:27.965 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.24414.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24414.log.gz 2024-09-18T12:04:27.978 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25017.log 2024-09-18T12:04:27.980 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.23794.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.23794.log.gz 2024-09-18T12:04:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.25017.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.25017.log.gz 2024-09-18T12:04:28.035 INFO:teuthology.orchestra.run.smithi129.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.admin.23773.log.gz 2024-09-18T12:04:28.218 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-18T12:04:28.278 INFO:teuthology.orchestra.run.smithi129.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.23975.log.gz 2024-09-18T12:04:28.541 INFO:teuthology.orchestra.run.smithi112.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-18T12:04:28.691 INFO:teuthology.orchestra.run.smithi129.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.23869.log.gz 2024-09-18T12:04:29.244 INFO:teuthology.orchestra.run.smithi112.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-18T12:04:29.245 INFO:teuthology.orchestra.run.smithi129.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-18T12:04:32.574 INFO:teuthology.orchestra.run.smithi112.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-18T12:04:32.754 INFO:teuthology.orchestra.run.smithi129.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-18T12:04:33.073 INFO:teuthology.orchestra.run.smithi112.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-18T12:04:33.199 INFO:teuthology.orchestra.run.smithi112.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-18T12:04:33.200 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-18T12:04:33.201 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m5.509s 2024-09-18T12:04:33.201 INFO:teuthology.orchestra.run.smithi112.stderr:user 0m17.473s 2024-09-18T12:04:33.201 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m0.665s 2024-09-18T12:04:33.560 INFO:teuthology.orchestra.run.smithi129.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-18T12:04:34.004 INFO:teuthology.orchestra.run.smithi129.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-18T12:04:34.006 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-18T12:04:34.006 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m6.267s 2024-09-18T12:04:34.006 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m19.806s 2024-09-18T12:04:34.007 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m0.764s 2024-09-18T12:04:34.007 INFO:tasks.ceph:Archiving logs... 2024-09-18T12:04:34.007 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/remote/smithi112/log 2024-09-18T12:04:34.008 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T12:04:37.008 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/remote/smithi129/log 2024-09-18T12:04:37.009 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T12:04:40.022 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T12:04:40.033 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T12:04:40.033 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T12:04:40.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T12:04:40.105 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T12:04:40.113 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T12:04:40.113 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:04:40.116 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:04:40.138 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T12:04:40.138 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2024-09-18T12:04:40.138 INFO:teuthology.orchestra.run.smithi129.stdout:*hv01.front.sepi 67.205.162.81 3 u 6 64 377 0.126 1.035 0.268 2024-09-18T12:04:40.138 INFO:teuthology.orchestra.run.smithi129.stdout:+hv02.front.sepi 63.231.80.2 3 u 63 64 377 0.072 2.041 0.321 2024-09-18T12:04:40.138 INFO:teuthology.orchestra.run.smithi129.stdout:+hv03.front.sepi 74.6.168.72 3 u 64 64 377 0.084 0.079 0.245 2024-09-18T12:04:40.138 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-18T12:04:40.139 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T12:04:40.140 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2024-09-18T12:04:40.140 INFO:teuthology.orchestra.run.smithi112.stdout:*hv01.front.sepi 67.205.162.81 3 u 66 64 377 0.079 -1.293 0.310 2024-09-18T12:04:40.140 INFO:teuthology.orchestra.run.smithi112.stdout:+hv02.front.sepi 63.231.80.2 3 u 62 64 377 0.073 -0.505 0.199 2024-09-18T12:04:40.140 INFO:teuthology.orchestra.run.smithi112.stdout:+hv03.front.sepi 74.6.168.72 3 u 65 64 377 0.090 -2.790 0.308 2024-09-18T12:04:40.140 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-18T12:04:40.140 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T12:04:40.148 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T12:04:40.148 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T12:04:40.155 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T12:04:40.163 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T12:04:40.169 INFO:teuthology.task.internal:Duration was 1727.347562 seconds 2024-09-18T12:04:40.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T12:04:40.177 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T12:04:40.178 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T12:04:40.184 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T12:04:40.237 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T12:04:40.237 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-09-18T12:04:40.237 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T12:04:40.249 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-09-18T12:04:40.249 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T12:04:40.261 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T12:04:40.262 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T12:04:40.264 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T12:04:40.298 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T12:04:40.298 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T12:04:40.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T12:04:40.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T12:04:40.419 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T12:04:40.419 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T12:04:40.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T12:04:40.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T12:04:40.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:04:40.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:04:40.470 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-09-18T12:04:40.505 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-09-18T12:04:40.521 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:04:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:04:40.548 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:04:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:04:40.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T12:04:40.563 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T12:04:40.563 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/remote/smithi112 2024-09-18T12:04:40.563 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T12:04:40.590 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910973/remote/smithi129 2024-09-18T12:04:40.590 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T12:04:40.613 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T12:04:40.613 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T12:04:40.615 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T12:04:40.656 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T12:04:40.664 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T12:04:40.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T12:04:40.673 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T12:04:40.673 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T12:04:40.675 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T12:04:40.681 INFO:teuthology.orchestra.run.smithi112.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 12:04 /home/ubuntu/cephtest 2024-09-18T12:04:40.683 INFO:teuthology.orchestra.run.smithi129.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 12:04 /home/ubuntu/cephtest 2024-09-18T12:04:40.685 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T12:04:40.694 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T12:04:40.713 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1727.347561597824 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T12:04:40.714 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T12:04:40.801 INFO:teuthology.run:pass